Lab 8 (Homography and Image Stitching)

Computer Vision I (CSCI 3240U)

Faisal Z. Qureshi

Faculty of Science, Ontario Tech University

http://vclab.science.ontariotechu.ca

Check Canvas for Due Date


Introduction

The goal of this lab is to stitch two photographs into one panorama using a homography. It brings the last three weeks together: SIFT descriptors propose correspondences, RANSAC rejects the wrong ones, and the direct linear transform (DLT) estimates the homography from what survives.

Parts 1 and 2 are required. Part 3 is optional and is not marked. The lab is sized for one two-hour session; budget up to two more hours at home if you need them.

Starter code is provided in stitching-starter.ipynb [html]. It loads the images, supplies a set of correspondences, and gives you a stitch helper, so your time goes into the parts that matter.

Reading

Computer Vision: Algorithms and Applications (2nd ed.), Szeliski, Sec. 8.1, 8.2.1.

Pairwise alignment; parametric motion models. The Week 9 slides cover everything this lab needs: the DLT, backward warping, the canvas, and the full pipeline. The ratio test is in the SIFT appendix of the Week 8 slides.

Data

The image pair boat1.jpg and boat2.jpg, supplied beside this handout. The scene is far from the camera, so parallax is negligible and a single homography relates the two views closely. The images are \(3888 \times 2592\); the starter works on a copy scaled by SCALE = 0.25 (\(972 \times 648\)), and every number in this handout refers to that working resolution.

Image credit: the “boat” stitching dataset from the OpenCV project’s test data, opencv_extra/testdata/stitching, used in OpenCV’s stitching tutorial.

Part 1: Homography from known correspondences

The starter supplies eight correspondences in PTS_LEFT and PTS_RIGHT, already checked. Row \(i\) of one list and row \(i\) of the other name the same point in the scene. Run the cell that draws them and convince yourself that they agree.

Task 1. Write homography_dlt(src, dst). It builds the \(2n \times 9\) matrix \(\mathbf{A}\) from \(n\) correspondences, solves \(\mathbf{A}\mathbf{h} = \mathbf{0}\) with the SVD, and returns \(\mathbf{H}\) scaled so that \(h_{33} = 1\). Estimate \(\mathbf{H}\) mapping the right image into the frame of the left, and check your answer against cv2.findHomography(src, dst, 0). They should agree closely but not exactly: OpenCV refines its estimate afterwards, so differences in the third decimal place are expected.

Task 2. Estimate \(\mathbf{H}\) twice, from the first four correspondences and from all eight. For each, report the reprojection error over all eight points (mean and maximum, in pixels). Explain what changes when you supply more than four points.

Careful: with exactly four points \(\mathbf{A}\) is \(8 \times 9\), and np.linalg.svd returns only eight singular values. The ninth, which is zero, is not in the list. The solution is still the last row of Vt.

Task 3. Stitch the pair with the given stitch(left, right, H) and show the result.

Part 2: Doing it automatically

Now replace the supplied correspondences with automatic ones.

Task 4. Detect SIFT keypoints and descriptors in both images with cv2.SIFT_create() and detectAndCompute(), on the greyscale working-resolution images. Match the descriptors with cv2.BFMatcher().knnMatch(des1, des2, k=2) and keep a match only if it passes Lowe’s ratio test with a ratio of 0.8. Report the number of keypoints in each image and the number of putative matches, and display a sample of the matches with cv2.drawMatches.

Task 5. Estimate the homography robustly with cv2.findHomography(src, dst, cv2.RANSAC, 3.0). The last argument is RANSAC’s inlier threshold \(d\), in pixels at the working resolution. Report the number of inliers and the inlier ratio, display the inlier matches only, and stitch with the resulting \(\mathbf{H}\).

Task 6. Answer both questions, with numbers and a figure.

Part 3 (optional)

None of these is marked.

Deliverables

Your notebook must contain the following.

Submission

Via Canvas. Please submit a single executed Jupyter notebook, one that has been run top to bottom, so that every figure and number listed above is visible in the submitted file. Code that has not been executed cannot be marked.

Parting thoughts