{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", "\n", "
\n", "\n", "# CSCI 3240U: Computer Vision I\n", "\n", "## Lab 8: Homography and Image Stitching\n", "\n", "Faisal Z. Qureshi \n", "Faculty of Science, Ontario Tech University \n", "Oshawa ON Canada \n", "\n", "\n", "*Fall 2026*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Copyright information\n", "\n", "© Faisal Qureshi" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## License\n", "\n", "\"Creative
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## About this notebook\n", "\n", "Read [lab8.md](lab8.md) first. This notebook covers the front of the lab only.\n", "It loads the image pair, holds your correspondences, and draws them so you can\n", "check them. Everything after that is yours to write.\n", "\n", "| Section here | Handout |\n", "|---|---|\n", "| Load the pair | Data |\n", "| Correspondences, typed in | Part 1, Tasks 1 and 2 |\n", "| Optional interactive point picker | Part 1, Tasks 1 and 2 |\n", "| Task 3 stub: estimate the homography | Part 1, Task 3 |\n", "| Task 4 stub: warp and stitch | Part 1, Task 4 |\n", "| Part 2 checklist | Part 2, Tasks 5 to 10 |\n", "\n", "Cells marked **TODO** are yours. Everything else runs as shipped.\n", "\n", "### Why the point picker is optional\n", "\n", "The picker that this notebook used to open with needed `%matplotlib tk`. That\n", "backend fails in JupyterLab and in Colab, so the notebook would not run at all\n", "for most of the class. The default path now needs no interactive backend: you\n", "type your picked coordinates into two lists. The picker is still here, one\n", "section further down, guarded by a flag. Turn it on only if you run this\n", "notebook on a desktop Jupyter with `%matplotlib tk`, or locally with\n", "`%matplotlib widget` after installing `ipympl`. Neither works in Colab.\n", "\n", "Any tool that reports pixel coordinates will do for the default path. Preview,\n", "GIMP and the Windows Photos app all show the cursor position, and so does the\n", "Matplotlib figure window when you run this notebook on your own machine." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "import cv2\n", "import numpy as np\n", "import matplotlib\n", "import matplotlib.pyplot as plt\n", "\n", "%matplotlib inline\n", "\n", "print('OpenCV ', cv2.__version__)\n", "print('NumPy ', np.__version__)\n", "print('Matplotlib', matplotlib.__version__)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Helper code\n", "\n", "Nothing below is graded. These helpers load images, display them, and draw\n", "labelled points. Read them before you use them.\n", "\n", "Two conventions matter and they are easy to mix up.\n", "\n", "- OpenCV point coordinates are `(x, y)`, that is `(column, row)`.\n", "- NumPy array indices are `[row, column]`.\n", "\n", "Every point array in this notebook is `(N, 2)` holding `(x, y)`, in float." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "DATA_FOLDER = '.'\n", "\n", "\n", "def load_rgb(filename):\n", " \"\"\"Read an image and return it as RGB, ready for plt.imshow.\n", "\n", " cv2.imread gives you BGR. Passing that straight to plt.imshow swaps red and\n", " blue, which is the classic first bug of every OpenCV lab. The version of\n", " this notebook shipped before Fall 2026 had exactly that bug.\n", " \"\"\"\n", " path = os.path.join(DATA_FOLDER, filename)\n", " bgr = cv2.imread(path)\n", " if bgr is None:\n", " raise FileNotFoundError(\n", " path + '\\n'\n", " 'Both images ship beside this notebook. On Colab, upload '\n", " '1-left.jpeg and 1-right.jpeg into the working directory, or set '\n", " 'DATA_FOLDER to wherever you put them.')\n", " return cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)\n", "\n", "\n", "def show(img, title=None, size=(10, 10), cmap=None):\n", " \"\"\"Display one image. Pass an RGB array, not a BGR one.\"\"\"\n", " plt.figure(figsize=size)\n", " if title:\n", " plt.title(title)\n", " if img.ndim == 2 and cmap is None:\n", " cmap = 'gray'\n", " plt.imshow(img, cmap=cmap)\n", " plt.axis('off')\n", " plt.show()\n", "\n", "\n", "def show_pair(img_a, img_b, title_a='', title_b='', size=(18, 8)):\n", " \"\"\"Display two RGB images side by side.\"\"\"\n", " plt.figure(figsize=size)\n", " for i, (img, title) in enumerate([(img_a, title_a), (img_b, title_b)]):\n", " plt.subplot(1, 2, i + 1)\n", " plt.title(title)\n", " plt.imshow(img)\n", " plt.axis('off')\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "\n", "def draw_points(img, pts, colour=(255, 0, 0), radius=None, thickness=None,\n", " label=True):\n", " \"\"\"Draw numbered circles on a copy of an RGB image.\n", "\n", " cv2.circle needs integer pixel coordinates. Handing it NumPy floats raises\n", " a TypeError in OpenCV 4.10 and later, so round and cast here.\n", " \"\"\"\n", " out = img.copy()\n", " h, w = out.shape[:2]\n", " if radius is None:\n", " radius = max(4, int(round(0.006 * max(h, w))))\n", " if thickness is None:\n", " thickness = max(2, radius // 3)\n", " for i, (x, y) in enumerate(np.asarray(pts, dtype=np.float64)):\n", " centre = (int(round(x)), int(round(y)))\n", " cv2.circle(out, centre, radius, colour, thickness)\n", " if label:\n", " cv2.putText(out, str(i), (centre[0] + radius, centre[1] - radius),\n", " cv2.FONT_HERSHEY_SIMPLEX, radius / 12.0, colour,\n", " thickness)\n", " return out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The image pair\n", "\n", "`1-left.jpeg` and `1-right.jpeg` are 4032 by 3024 pixels each. Full resolution\n", "is slow and wastes memory, and a Colab session will not thank you for a warped\n", "canvas of that size, so everything below works on a scaled copy.\n", "\n", "`SCALE` sets the working resolution. Keep your correspondences in\n", "**full resolution pixel coordinates** and let the code scale them, so that your\n", "numbers stay valid when you change `SCALE`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "SCALE = 0.25 # working resolution; set to 1.0 for the full 4032 x 3024 images\n", "\n", "left_full = load_rgb('1-left.jpeg')\n", "right_full = load_rgb('1-right.jpeg')\n", "\n", "left = cv2.resize(left_full, None, fx=SCALE, fy=SCALE,\n", " interpolation=cv2.INTER_AREA)\n", "right = cv2.resize(right_full, None, fx=SCALE, fy=SCALE,\n", " interpolation=cv2.INTER_AREA)\n", "\n", "print('full resolution ', left_full.shape, right_full.shape)\n", "print('working resolution', left.shape, right.shape)\n", "\n", "show_pair(left, right, '1-left.jpeg', '1-right.jpeg')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Tasks 1 and 2: your correspondences\n", "\n", "Fill in the two lists below. Each row is one point, written as `[x, y]` in\n", "**full resolution** pixels. Row $i$ of `PTS_LEFT` and row $i$ of `PTS_RIGHT`\n", "must name the same physical point in the scene. Order is everything: a pair of\n", "lists that disagree on order is the most common way to get a nonsensical\n", "homography, and the code cannot detect it for you.\n", "\n", "The values below are an **example**, not an answer. They let the notebook run\n", "before you have picked anything. Replace them with your own picks, and pick\n", "more than four so that Part 1 can also look at the overdetermined case.\n", "\n", "Where to pick, on this pair: the two photographs overlap over roughly the right\n", "quarter of the left image and the left quarter of the right image. Look at the\n", "window, the blind and the sill, the plant and its white pot, and the lamp arm.\n", "Nothing on the far left of the left image or the far right of the right image\n", "appears in both, so there is nothing there to pick.\n", "\n", "Spread your points out. Four points crowded into one corner pin the homography\n", "there and let it run wild everywhere else. Depth matters too: most of the\n", "example points below sit on the plant, which is the nearest thing in the scene\n", "and therefore the worst affected by the camera moving between the two shots.\n", "Picking on the window and the sill instead is worth trying, and comparing the\n", "two homographies is worth a paragraph in your write-up." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Example correspondences, in FULL RESOLUTION pixels. Replace with your own.\n", "# Each row is [x, y]. Row i of PTS_LEFT matches row i of PTS_RIGHT.\n", "PTS_LEFT = [\n", " [3097, 798], # 0 on the blind, high in the overlap\n", " [3662, 1861], # 1 leaf, upper part of the hanging plant\n", " [3321, 2234], # 2 leaf against the window screen\n", " [3695, 2125], # 3 leaf and stem above the white pot\n", " [3136, 2490], # 4 leaf low on the left of the overlap\n", " [3966, 2576], # 5 white pot, low on the right of the overlap\n", "]\n", "\n", "PTS_RIGHT = [\n", " [ 90, 846], # 0\n", " [ 705, 1841], # 1\n", " [ 400, 2259], # 2\n", " [ 761, 2097], # 3\n", " [ 210, 2568], # 4\n", " [1045, 2487], # 5\n", "]\n", "\n", "pts_left = np.asarray(PTS_LEFT, dtype=np.float64) * SCALE\n", "pts_right = np.asarray(PTS_RIGHT, dtype=np.float64) * SCALE\n", "\n", "assert pts_left.shape == pts_right.shape, 'the two lists must be the same length'\n", "assert pts_left.shape[0] >= 4, 'a homography needs at least four correspondences'\n", "assert pts_left.ndim == 2 and pts_left.shape[1] == 2, 'each row must be [x, y]'\n", "\n", "print(len(pts_left), 'correspondences at the working resolution')\n", "print(np.round(np.hstack([pts_left, pts_right]), 1))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Check your correspondences before you use them\n", "\n", "Look at the two figures below and read the numbers off. Point 3 on the left and\n", "point 3 on the right must sit on the same thing in the scene. Fix any that do\n", "not before you go any further. Debugging a homography whose input is wrong\n", "wastes an afternoon." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "show_pair(draw_points(left, pts_left, colour=(255, 0, 0)),\n", " draw_points(right, pts_right, colour=(255, 0, 0)),\n", " 'left: picked points', 'right: matching points')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Optional: the interactive point picker\n", "\n", "**This section needs an interactive Matplotlib backend and does not run in\n", "Colab.** Skip it there and type your coordinates into the lists above.\n", "\n", "To use it:\n", "\n", "- on a desktop Jupyter, put `%matplotlib tk` in the cell below;\n", "- locally in JupyterLab, `pip install ipympl` and use `%matplotlib widget`;\n", "- then set `RUN_PICKER = True` and run the cell.\n", "\n", "Under `%matplotlib inline`, which is what this notebook selects, the figure is a\n", "static image and no mouse events reach Python, so the picker collects nothing.\n", "The flag keeps that from looking like a bug.\n", "\n", "Click a point and the picker prints it, marks it, and appends it to\n", "`picker.points`. When you are done, run the cell after it to print your points\n", "in the format the lists above expect, then paste them in. That way your picks\n", "survive a restart, and the graders can see them.\n", "\n", "
\n", "\n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "RUN_PICKER = False # set to True only with an interactive backend\n", "\n", "# %matplotlib tk # desktop Jupyter\n", "# %matplotlib widget # JupyterLab with ipympl installed\n", "\n", "\n", "class PointPicker:\n", " \"\"\"Collect (x, y) locations from mouse clicks on a Matplotlib image.\n", "\n", " The picked points are kept in self.points as (x, y) pairs in the\n", " coordinates of the array you passed in. Multiply by 1 / SCALE to get back\n", " to full resolution pixels.\n", " \"\"\"\n", "\n", " def __init__(self, imgplot, img, colour=(255, 0, 0), radius=8, thickness=3):\n", " self.imgplot = imgplot\n", " self.img = img.copy()\n", " self.colour = colour\n", " self.radius = radius\n", " self.thickness = thickness\n", " self.points = []\n", " self.cid = imgplot.figure.canvas.mpl_connect('button_press_event', self)\n", "\n", " def __call__(self, event):\n", " if event.inaxes != self.imgplot.axes:\n", " return\n", " if event.xdata is None or event.ydata is None:\n", " return\n", " x, y = float(event.xdata), float(event.ydata)\n", " self.points.append((x, y))\n", " print('%d: x=%.1f, y=%.1f' % (len(self.points) - 1, x, y))\n", "\n", " # cv2.circle needs integer coordinates; floats raise a TypeError.\n", " cv2.circle(self.img, (int(round(x)), int(round(y))), self.radius,\n", " self.colour, self.thickness)\n", " self.imgplot.set_array(self.img)\n", " self.imgplot.figure.canvas.draw()\n", "\n", " def as_array(self, scale=1.0):\n", " \"\"\"Return the picks as an (N, 2) float array, divided by scale.\"\"\"\n", " return np.asarray(self.points, dtype=np.float64) / scale\n", "\n", "\n", "if RUN_PICKER:\n", " picker_image = left # or right\n", " fig = plt.figure(figsize=(10, 10))\n", " ax = fig.add_subplot(111)\n", " imgplot = ax.imshow(picker_image) # RGB, so the colours are right\n", " picker = PointPicker(imgplot, picker_image)\n", " plt.show()\n", "else:\n", " print('Picker off. Type your coordinates into PTS_LEFT and PTS_RIGHT.')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Run this after picking, then paste the output into the lists above.\n", "if RUN_PICKER and picker.points:\n", " for x, y in picker.as_array(scale=SCALE):\n", " print(' [%4d, %4d],' % (round(x), round(y)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Task 3: estimate the homography\n", "\n", "From here on the code is yours. The handout allows you to start with\n", "`cv2.findHomography` so that you have something working, and then requires your\n", "own implementation. Write both and compare the two matrices.\n", "\n", "Recall the shape of the problem. Each correspondence contributes two rows to\n", "$\\mathbf{A}$, a homography has 8 degrees of freedom, and you solve\n", "$\\mathbf{A}\\mathbf{h} = \\mathbf{0}$ with the SVD, exactly as in Lab 6. Then\n", "reshape $\\mathbf{h}$ into a $3 \\times 3$ matrix.\n", "\n", "Decide which direction your $H$ maps, write it in the docstring, and keep it\n", "straight. Half of the failures in this lab are a matrix applied backwards." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def estimate_homography(src_pts, dst_pts):\n", " \"\"\"Estimate H mapping src_pts to dst_pts. Both are (N, 2) arrays of (x, y).\n", "\n", " Returns a 3x3 matrix.\n", " \"\"\"\n", " # TODO (Task 3)\n", " # 1. build A, two rows per correspondence\n", " # 2. solve A h = 0 with the SVD\n", " # 3. reshape h to 3x3 and normalise\n", " raise NotImplementedError('Task 3: estimate_homography')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Task 4: warp and stitch\n", "\n", "Warp one image into the frame of the other and put the two on one canvas.\n", "\n", "Two details decide whether this works.\n", "\n", "- **Warp backwards.** For each destination pixel, apply the inverse mapping to\n", " find where it came from in the source, then interpolate there. Forward\n", " warping scatters source pixels into the destination and leaves a lattice of\n", " holes. The handout warns about this and the warning is worth heeding.\n", "- **Size the canvas first.** Map the four corners of the image you are warping\n", " into the destination frame. Their bounding box, unioned with the destination\n", " image, tells you how large the canvas must be and how far to translate\n", " everything so that nothing lands at a negative coordinate.\n", "\n", "On this pair the two photographs overlap over about a quarter of their width,\n", "and the camera translated rather than rotated, so expect a visibly distorted\n", "result with ghosting on the near objects. That is the parallax the handout\n", "asks you to discuss, not a bug in your code. Crop the canvas before you display\n", "it if it comes out very large." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def warp_backward(img, H, out_shape):\n", " \"\"\"Warp img into an out_shape canvas using backward mapping.\n", "\n", " out_shape is (height, width). Returns an array of that shape.\n", " \"\"\"\n", " # TODO (Task 4)\n", " # 1. build the grid of destination pixel coordinates\n", " # 2. map them into the source with the inverse of H\n", " # 3. interpolate the source at those locations, bilinear is expected\n", " # 4. leave destination pixels whose source falls outside the image alone\n", " raise NotImplementedError('Task 4: warp_backward')\n", "\n", "\n", "def stitch(img_left, img_right, H):\n", " \"\"\"Place both images on one canvas and return it.\"\"\"\n", " # TODO (Task 4)\n", " # 1. map the corners of the warped image into the reference frame\n", " # 2. work out the canvas size and the translation that keeps it positive\n", " # 3. warp, then paste the reference image in\n", " raise NotImplementedError('Task 4: stitch')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Then: more than four points\n", "\n", "Repeat Task 3 with more correspondences and report what changes. The handout\n", "asks you to show it, so show a figure or a table, not a sentence. Watch the\n", "smallest singular value of $\\mathbf{A}$ as you add points, and say what it\n", "measures." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Part 2: doing it automatically\n", "\n", "No starter code here. Tasks 5 to 10 are yours, and the handout lists them. The\n", "pieces you need already exist in earlier labs.\n", "\n", "- Detect and describe with `cv2.SIFT_create()`. SIFT moved into the main\n", " OpenCV module in version 4.4. `cv2.xfeatures2d.SIFT_create()` is gone from\n", " `opencv-python` and raises an `AttributeError`.\n", "- Match with `cv2.BFMatcher` and `knnMatch`, then apply the ratio test.\n", "- Run RANSAC over the putative matches, as in Lab 6, and report the inlier\n", " count.\n", "- Justify your iteration count from the inlier ratio you observed. Quoting a\n", " default of 1000 earns nothing.\n", "- Show the no-RANSAC failure: least squares over every putative match, outliers\n", " included.\n", "\n", "One warning about this particular pair. The overlap is small, the scene is\n", "indoors, and the blind fills much of the frame with a repeating pattern, so\n", "plain SIFT matching on it proposes many wrong matches and the inlier ratio comes\n", "out low. Report the ratio you measure and let it drive your iteration count.\n", "If your automatic result is poor here, say so and explain why, using the\n", "putative match figure as evidence. A pair of your own, shot by rotating in\n", "place, will behave much better." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Submission\n", "\n", "Submit a single Jupyter notebook **via Canvas**. Run it top to bottom before\n", "you submit, so every figure and every number is visible in the file. Code that\n", "has not been executed cannot be marked." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " \n", " \n", " \n", "
" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.0" } }, "nbformat": 4, "nbformat_minor": 4 }