Computer Vision I (CSCI 3240U)
Faculty of Science, Ontario Tech University
http://vclab.science.ontariotechu.ca
Check Canvas for Due Date
The goal of this lab is to stitch two photographs into one panorama using a homography. It brings the last three weeks together: SIFT descriptors propose correspondences, RANSAC rejects the wrong ones, and the direct linear transform (DLT) estimates the homography from what survives.
Parts 1 and 2 are required. Part 3 is optional and is not marked. The lab is sized for one two-hour session; budget up to two more hours at home if you need them.
Starter code is provided in stitching-starter.ipynb [html]. It loads the images, supplies a
set of correspondences, and gives you a stitch helper, so
your time goes into the parts that matter.
Computer Vision: Algorithms and Applications (2nd ed.), Szeliski, Sec. 8.1, 8.2.1.
Pairwise alignment; parametric motion models. The Week 9 slides cover everything this lab needs: the DLT, backward warping, the canvas, and the full pipeline. The ratio test is in the SIFT appendix of the Week 8 slides.
The image pair boat1.jpg and boat2.jpg, supplied beside this handout. The scene
is far from the camera, so parallax is negligible and a single
homography relates the two views closely. The images are \(3888 \times 2592\); the starter works on a
copy scaled by SCALE = 0.25 (\(972 \times 648\)), and every number in this
handout refers to that working resolution.
Image credit: the “boat” stitching dataset from the OpenCV
project’s test data, opencv_extra/testdata/stitching,
used in OpenCV’s stitching tutorial.
The starter supplies eight correspondences in PTS_LEFT
and PTS_RIGHT, already checked. Row \(i\) of one list and row \(i\) of the other name the same point in the
scene. Run the cell that draws them and convince yourself that they
agree.
Task 1. Write homography_dlt(src, dst).
It builds the \(2n \times 9\) matrix
\(\mathbf{A}\) from \(n\) correspondences, solves \(\mathbf{A}\mathbf{h} = \mathbf{0}\) with
the SVD, and returns \(\mathbf{H}\)
scaled so that \(h_{33} = 1\). Estimate
\(\mathbf{H}\) mapping the right image
into the frame of the left, and check your answer against
cv2.findHomography(src, dst, 0). They should agree closely
but not exactly: OpenCV refines its estimate afterwards, so differences
in the third decimal place are expected.
Task 2. Estimate \(\mathbf{H}\) twice, from the first four correspondences and from all eight. For each, report the reprojection error over all eight points (mean and maximum, in pixels). Explain what changes when you supply more than four points.
Careful: with exactly four points \(\mathbf{A}\) is \(8 \times 9\), and
np.linalg.svd returns only eight singular values. The
ninth, which is zero, is not in the list. The solution is still the last
row of Vt.
Task 3. Stitch the pair with the given
stitch(left, right, H) and show the result.
Now replace the supplied correspondences with automatic ones.
Task 4. Detect SIFT keypoints and descriptors in
both images with cv2.SIFT_create() and
detectAndCompute(), on the greyscale working-resolution
images. Match the descriptors with
cv2.BFMatcher().knnMatch(des1, des2, k=2) and keep a match
only if it passes Lowe’s ratio test with a ratio of
0.8. Report the number of keypoints in each image and
the number of putative matches, and display a sample of the matches with
cv2.drawMatches.
Task 5. Estimate the homography robustly with
cv2.findHomography(src, dst, cv2.RANSAC, 3.0). The last
argument is RANSAC’s inlier threshold \(d\), in pixels at the working resolution.
Report the number of inliers and the inlier ratio, display the inlier
matches only, and stitch with the resulting \(\mathbf{H}\).
Task 6. Answer both questions, with numbers and a figure.
cv2.findHomography(src, dst, 0), and stitch with it. Show
the result and explain it.None of these is marked.
cv2.warpPerspective with your own backward warp and
bilinear interpolation. For each canvas pixel, apply \(\mathbf{H}^{-1}\) to find where it came
from, then interpolate.cv2.RANSAC
with the RANSAC you wrote in Lab 6, using homography_dlt on
four-point samples.Your notebook must contain the following.
homography_dlt, and its \(\mathbf{H}\) compared against
OpenCV’s.Via Canvas. Please submit a single executed Jupyter notebook, one that has been run top to bottom, so that every figure and number listed above is visible in the submitted file. Code that has not been executed cannot be marked.