Introduction

The goal of this lab is to stitch two images together using a homography. This is the lab where the last three weeks come together: you will use the interest points and descriptors from Lab 7 to propose correspondences, RANSAC from Lab 6 to reject the wrong ones, and least squares to estimate the homography from what survives.

Parts 1 and 2 are required. Part 3 is optional — attempt it if you have time, but it will not be marked.

Starter code for picking points and for the basic stitching pipeline is provided in stitching-starter.ipynb [html].

Reading

Computer Vision: Algorithms and Applications (2nd ed.), Szeliski — Sec. 8.1, 8.2.1.

Pairwise alignment; parametric motion models.

Data

The image pair 1-left.jpeg and 1-right.jpeg, supplied beside this handout. Part 3 asks for a pair you photograph yourself.

Part 1: Homography from hand-picked correspondences

Start with correspondences you select yourself, so that you can debug the geometry without worrying about bad matches.

Task 1. Load the left image (1-left.jpeg) and pick at least four points in order.

Task 2. Load the right image (1-right.jpeg) and pick the corresponding locations in the same order.

Task 3. Use the correspondences to estimate the homography.

Task 4. Use the homography to warp the second image into the frame of the first and stitch them together.

It is possible to complete this part using the built-in OpenCV functions findHomography and warpPerspective, and it is a good idea to start there so that you have something working. You must then implement the functionality yourself:

• Setting up the system of linear equations for the homography;
• Solving \(\mathbf{A}\mathbf{x} = \mathbf{0}\) (recall from Lab 6 how to do this with the SVD);
• Warping the image using the estimated homography; and
• Estimating pixel colours in the destination image using interpolation.

Recall that a homography has 8 degrees of freedom, so 4 point correspondences is the minimum. What happens when you supply more than 4? Show it.

Part 2: Doing it automatically

Now replace the human in the loop.

Task 5. Detect interest points in both images and compute descriptors (Lab 7).

Task 6. Match the descriptors to propose a set of putative correspondences.

Task 7. Display the putative matches. How many of them are obviously wrong?

Task 8. Use RANSAC (Lab 6) to estimate the homography robustly from these putative matches.

Task 9. Report the number of inliers and display the inlier correspondences only.

Task 10. Stitch using the resulting homography.

Compare your automatic result against your hand-picked result from Part 1. Which is better, and why?

Task 11: questions to answer

• How many RANSAC iterations did you use? Justify the number using the formula from class, given your observed inlier ratio.
• What happens to the estimated homography if you skip RANSAC and simply run least squares on all putative matches? Show the result — it is instructive.

Part 3 (optional): Your own panorama

Go for a walk and take an image pair of your favourite neighbourhood landscape. Try not to translate the camera — just rotate in place. Stitch your own pair.

The provided image pair exhibits strong parallax. Explain what parallax does to your stitching result, and why rotating in place avoids the problem.

Deliverables

Your notebook must contain the following.

• Your hand-picked correspondences and the resulting homography.
• Your own implementation: the linear system, the SVD solution, backward warping, and interpolation — not only the OpenCV version.
• The stitched image, and what you observed when supplying more than four points.
• The putative matches from automatic matching, displayed.
• The RANSAC inlier count and a figure showing inliers only, plus the automatic stitch.
• A RANSAC iteration count justified from your observed inlier ratio, not a default value.
• The no-RANSAC failure figure — least squares over all putative matches.

Submission

Via Canvas. Please submit a single executed Jupyter notebook — one that has been run top to bottom, so that every figure and number listed above is visible in the submitted file. Code that has not been executed cannot be marked.

Parting thoughts

• User interaction is a big piece of Part 1. How will the user select corresponding points in two images? What if they make a mistake — do they have to start from the beginning? It is worth thinking about the experience you want to provide. Alas, OpenCV cannot help you there.
• Blending the two images where they overlap is not required, but a simple feathered blend makes an enormous difference to how the result looks.
• If your warped image comes out mostly black, check whether you are warping forward or backward. Backward warping — for each destination pixel, find where it came from in the source — is almost always what you want.