Introduction

The goal of this lab is to recover depth from a pair of images, and to check that the depths you recover are physically believable.

Every lab so far has worked in the image plane. This one leaves it. By the end you will be able to point at a car in a KITTI frame and say how many metres away it is — and, more importantly, say how much you trust that number.

The KITTI stereo pair is rectified, which means corresponding points lie on the same image row. That reduces the correspondence search from two dimensions to one, and it is the single most useful thing epipolar geometry does for you.

Reading

Computer Vision: Algorithms and Applications (2nd ed.), Szeliski — Sec. 12.1–12.5.

Epipolar geometry, rectification, sparse and dense correspondence, local methods and global optimization.

Data

You need the right camera images for this lab, which the earlier labs did not require. If you ran fetch-kitti.sh --left-only, run it again without the flag:

$ ./fetch-kitti.sh

See Getting the KITTI data. Left images are in image_2/, right images in image_3/, and each frame has a calibration file in calib/.

A note on loading the data

Parsing KITTI’s calibration format and decoding its colour-coded ground truth is plumbing, not computer vision, and debugging it eats time you should be spending on the lab. A small helper module is provided: kitti.py.

import kitti

ds  = kitti.Dataset("path/to/data_road")     # point at the extracted folder
img = ds.image("um_000032")                  # left image, RGB uint8
cal = ds.calib("um_000032")                  # cal.K, cal.fx, cal.baseline, ...
pos, valid = ds.ground_truth("um_000032")    # boolean masks

It deliberately does not implement anything you are asked to write yourself — no edge detection, no region of interest, no scoring. You are free to ignore it and do your own file handling if you prefer.

Part 1: Reading the calibration

Each calib/*.txt contains projection matrices P0–P3. P2 is the left colour camera and P3 is the right. Each is \(3 \times 4\): \[ P = \left[\begin{array}{cc} K & K\mathbf{t} \end{array}\right], \] where \(K\) holds the intrinsics and \(\mathbf{t}\) the translation relative to the reference camera.

Task 1

Parse P2 and P3 for one frame and extract:

• the focal length \(f\) in pixels (the \((0,0)\) entry),
• the principal point \((c_x, c_y)\),
• the stereo baseline \(B\) in metres.

For the baseline, note that the last column of \(P\) is \(K\mathbf{t}\), so the horizontal translation is \(P[0,3]/f\). The baseline is the difference between the two cameras’ horizontal translations: \[ B = \frac{|P_3[0,3] - P_2[0,3]|}{f}. \]

Report your numbers. KITTI’s cameras are roughly half a metre apart and the focal length is a few hundred pixels; if you get a baseline of 400 or 0.0005, you have forgotten to divide by \(f\) or divided twice.

Task 2

Confirm that the two cameras really are rectified: pick several corresponding points by hand in the left and right images and check that their \(y\) coordinates agree to within a pixel or two. Report the residual vertical disparities.

Part 2: Correspondence by block matching

Task 3: implement it

For each pixel in the left image, search along the same row in the right image for the best matching window. Because the right camera is to the right of the left one, the match always appears at a smaller or equal \(x\), so search \(d \in [0, d_{\max}]\) where \(x_{\text{right}} = x_{\text{left}} - d\).

Implement at least two matching costs:

• SSD — sum of squared differences.
• NCC — normalised cross-correlation.

Produce a disparity map for one frame with each cost and display them with a sensible colour map.

Task 4: window size

Vary the block size (say \(3, 7, 15, 31\)) and show the results. Describe the trade-off you observe — what improves and what degrades as the window grows. Pay particular attention to what happens at object boundaries and on the road surface.

Task 5: where does it fail?

Identify at least three regions in your disparity map that are clearly wrong, and name the cause for each. Candidates to look for in these scenes:

• textureless road and sky, where every window matches equally well,
• repeated structure such as railings and fences,
• surfaces visible in one camera but occluded in the other,
• specular or saturated regions.

Show crops. A claim without a crop is not evidence.

Task 6: a consistency check

You have no ground truth here, so you must validate your own output.

Compute disparity twice — once matching left-to-right and once right-to-left — and keep only pixels where the two agree to within one pixel. Report the fraction of pixels that survive, and display the map with the failures masked out.

Compare against cv.StereoBM and cv.StereoSGBM on the same frame. Report the fraction of pixels each of them labels valid. You should find SGBM produces a substantially denser map than BM; explain what SGBM adds that plain block matching does not.

Part 3: From disparity to metres

Task 7: triangulate

For a rectified pair, \[ Z = \frac{f B}{d}, \] where \(Z\) is depth in metres, \(f\) the focal length in pixels, \(B\) the baseline in metres, and \(d\) the disparity in pixels.

Convert your disparity map to a depth map. State the depth range you recover and sanity-check it: the near end should be a few metres, the far end should be somewhere in the tens to low hundreds of metres.

Task 8: how good is a far measurement?

Differentiate the triangulation equation to show that \[ \left|\frac{\partial Z}{\partial d}\right| = \frac{fB}{d^2} = \frac{Z^2}{fB}. \]

So depth error grows with the square of depth.

Using your measured \(f\) and \(B\), tabulate the depth uncertainty corresponding to a half-pixel disparity error at \(Z = 5, 10, 20, 50\) and \(100\) m.

Then answer: if you were building an emergency braking system, at what range would you stop trusting this sensor, and what would you change to push that range further out?

Task 9: validate against the road plane

The calibration file contains Tr_cam_to_road, the transform from the camera to a plane fitted to the road surface. This gives you a free ground-truth check.

Take pixels inside the road region (use the Lab 4 ground-truth mask), convert them to 3D points using your depth map, and check how far they lie from the road plane. Report the mean and 95th-percentile deviation in metres.

This is the strongest evidence in the lab that your depths are real rather than merely plausible.

Part 4 (optional): Epipolar geometry without rectification

Everything above relied on the images being rectified. When they are not, you need the fundamental matrix \(F\).

Take two images of the same scene from your phone, from different viewpoints. Find correspondences using your Lab 7 descriptors, estimate \(F\) with the eight-point algorithm inside RANSAC (Lab 6), and draw the epipolar lines.

Verify that corresponding points lie on each other’s epipolar lines, and report the mean point-to-line distance.

Deliverables

Your notebook must contain the following.

• A table of \(f\), the principal point and the baseline in metres, plus your rectification check residuals.
• Disparity maps from both SSD and NCC.
• A window-size study across at least four block sizes.
• Three failure regions, each with a crop and a named cause.
• The left-right consistency check: the fraction of pixels surviving, and the masked disparity map.
• Valid-pixel fractions for StereoBM and StereoSGBM, with an explanation of what SGBM adds.
• A depth map in metres and the depth range you recover.
• The uncertainty table for a half-pixel disparity error at \(Z = 5, 10, 20, 50, 100\) m, derived from \(\partial Z/\partial d\).
• The road-plane validation: mean and 95th-percentile deviation in metres, linked back to the \(Z^2\) law.

Submission

Via Canvas. Please submit a single executed Jupyter notebook — one that has been run top to bottom, so that every figure and number listed above is visible in the submitted file. Code that has not been executed cannot be marked.

Parting thoughts

• Write the block matcher yourself. A naive triple loop over pixels and disparities will be slow; compute the cost volume with array operations instead, one disparity at a time, and take the argmin at the end.
• Disparity of zero means infinitely far away. Guard your division.
• OpenCV’s stereo classes return fixed-point disparities scaled by 16. Divide by 16 before doing anything physical with them. Forgetting this is the most common reason a depth map comes out 16 times too close.
• Sub-pixel disparity, obtained by fitting a parabola through the cost at the winning disparity and its two neighbours, is a few lines of code and visibly reduces the “layered” look of the depth map. Try it.
• Notice how much of this lab is about not trusting the output. That is not pessimism; a depth sensor that does not know when it is wrong is a dangerous thing to put in a vehicle.