Table of Contents:

PatchMatch

“some good patch matches can be found via random sampling, and that natural coherence in the imagery allows us to propagate such matches quickly to surrounding areas” [PatchMatch paper]

“the inherent spatial coherence of depth maps is exploited to quickly find a good solution without the need to look through all possibilities” [PatchMatch Net]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (SIGGRAPH), 2009

paper: https://gfx.cs.princeton.edu/pubs/Barnes_2009_PAR/patchmatch.pdf

Dynamic programming, used in this case. Use a single match as constraint to find other matches. get ordering constraint.

sometimes blue patch is not going to happen! happens all the time. have to allow that sometime there will not be a match, and just move on to the next guy.

real-time stereo in late 80s from stereo matching with dynamic programming.

Turns out there are quite a few problems.

illusion? thin nail illusion, thin object close to the camera?

slanted plane: scrunched, will have multiple matches in one location.

one scanline at a time, wasnt the winning approach.

matching the scannline independently isn’t good – may see weird artifacts when you look at a vertical column, 3d boundaries in scene may not make straight lines. no consistently through your scanlines.

Scanline is bad as representation of the world. segemntation based stereo – first segment the iamge into patches that are similar in their appearance, then work patch by patch and do the matching.

you can also get view interpolation, synthesize new views in between the two views.

Simple Example on KITTI

\[E = R \mbox{ } [t]_{\times}\]

Object Detection in Stereo

Chen et al. 3DOP (NIPS 2015) Urtasun [4]

Unsupervised Learning of Depth

From 2017 [5]

Let \(p_t\) denote homogeneous coordinates in the target view. \(K\) denotes the camera intrinsic matrix.

\[p_s \sim K \hat{T}_{t \rightarrow s} \hat{D}_t (p_t) K^{-1} p_t\]

[4] X. Chen, K. Kundu, Y. Zhu, A. Berneshawi, H. Ma, S. Fidler, R. Urtasun. 3D Object Proposals for Accurate Object Class Detection. Advances in Neural Information Processing Systems 28 (NIPS 2015). PDF.