CSCI 3240U: Computer Vision I

Lab 10: Convolutional Networks and Pretrained Features

Faisal Z. Qureshi
Faculty of Science, Ontario Tech University
Oshawa ON Canada
http://vclab.science.ontariotechu.ca

Fall 2026

Copyright information

© Faisal Qureshi

License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

About this notebook

This notebook is the starter code for Lab 10. It follows the lab handout task by task. Each section below carries the task number from the handout, so read the handout first and keep it open beside this notebook.

Most of the code is provided. Your job is to run it, fill in the two places marked TODO, and write the discussion the handout asks for.

The handout, not this notebook, lists what you must submit. See the Deliverables section of lab10. Submit a single notebook that has been executed top to bottom.

We take a ResNet trained on ImageNet, throw away its 1000-way classifier, bolt a 10-way classifier onto the frozen backbone, and train only that. Then we do the thing that gives the lab its point: we train the same linear classifier on raw pixels, on a feature you designed by hand in Lab 4, and on the pretrained network's features, and compare all three.

You do not need a GPU. The backbone stays frozen, so the only thing being trained is one linear layer.

We use a subsample of CIFAR10. Running a ResNet over all 60,000 images at 224x224 on a CPU takes far too long for a lab session, so we draw a random subset. The sizes are the variables N_TRAIN and N_TEST in the setup cell. Raise them if you have time or a GPU, and say in your write-up what you used.

Setup

Everything this notebook needs is in the course environment (see Lab 1). No pip install is required.

import time

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import torch
import torch.nn as nn
import torchvision
from torch.utils.data import DataLoader, Subset
from torchvision import models, transforms

from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression

%matplotlib inline

# ==== configuration.  Change these, do not scatter magic numbers below. =======

SEED = 42

DATA_ROOT = './data'      # CIFAR10 is downloaded here the first time you run

IMG_SIZE = 224            # what the pretrained ResNet expects.  See Task 3.
N_TRAIN = 5000            # subsample of the 50,000 CIFAR10 training images
N_TEST = 1000             # subsample of the 10,000 CIFAR10 test images
N_PCA = 500               # how many test images to plot in Task 5

BATCH_SIZE = 64
NUM_WORKERS = 0           # set to 2 on Colab for faster loading
EPOCHS = 15               # epochs for the linear head
LR = 1e-3
MAX_ITER = 500            # iterations for scikit-learn's logistic regression

# ImageNet channel statistics.  These are a property of the pretrained model,
# not of CIFAR10.  Do not replace them with CIFAR10's own mean and std.
IMAGENET_MEAN = [0.485, 0.456, 0.406]
IMAGENET_STD = [0.229, 0.224, 0.225]

torch.manual_seed(SEED)
np.random.seed(SEED)

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
print('device:', device)

Task 1: get a pretrained model

We load ResNet-18 with its ImageNet weights. ResNet-18 is the smallest of the family and is plenty for this lab.

The handout points you at timm, which is an excellent library and worth knowing. We use torchvision here because it ships with PyTorch and needs no extra install. The timm equivalent is one line:

import timm
resnet = timm.create_model('resnet18', pretrained=True)

Either route is acceptable. Everything below assumes the torchvision model.

weights = models.ResNet18_Weights.IMAGENET1K_V1
resnet = models.resnet18(weights=weights)
resnet.eval()

print(resnet)

Scroll to the bottom of that printout. The last two lines are the interesting ones:

• avgpool, which collapses each of the 512 final feature maps to one number. Its output is the penultimate representation, a 512-dimensional vector per image. We use it again in Tasks 4 and 5.
• fc, a Linear(512, 1000). That is the classifier head, and it is the part we replace.

The weights object also records how the model was trained. Read the input size and the normalisation off it rather than guessing.

print('classifier head :', resnet.fc)
print('feature size    :', resnet.fc.in_features)
print()
print('preprocessing the weights were trained with:')
print(weights.transforms())

Task 2: swap the head and freeze the backbone

Two steps, in this order.

1. Set requires_grad = False on every parameter, which freezes the whole network.
2. Replace fc with a fresh Linear(512, 10). A newly created layer has requires_grad = True, so the head trains and nothing else does.

Do it the other way round and you freeze your own head as well.

The parameter count is the check that tells you it worked. A Linear(512, 10) holds $512 \times 10$ weights plus 10 biases, which is $5130$ numbers. If the printout below says anything else, the freeze went wrong. Compare against the frozen count: roughly 11 million. That ratio is why this lab runs on a laptop.

NUM_CLASSES = 10

num_features = resnet.fc.in_features          # 512 for ResNet-18

for param in resnet.parameters():             # 1. freeze everything
    param.requires_grad = False

resnet.fc = nn.Linear(num_features, NUM_CLASSES)   # 2. fresh, trainable head

model = resnet.to(device)

trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in model.parameters() if not p.requires_grad)
expected = num_features * NUM_CLASSES + NUM_CLASSES

print(f'penultimate feature size : {num_features}')
print(f'trainable parameters     : {trainable:,}')
print(f'frozen parameters        : {frozen:,}')
print(f'trainable fraction       : {trainable / (trainable + frozen):.5%}')
print()
print(f'head should hold {num_features} x {NUM_CLASSES} weights + {NUM_CLASSES} '
      f'biases = {expected:,}')

assert trainable == expected == 5130, 'the freeze or the head is wrong'

Task 3: train the head and evaluate

Preprocessing, which is the part that catches people

CIFAR10 images are 32x32, and they are stored as 8-bit RGB. The pretrained ResNet was trained on 224x224 crops of ImageNet, normalised with ImageNet's channel mean and standard deviation. Feed it 32x32 images, or normalise with CIFAR10's own statistics, and the accuracy collapses. The features the backbone computes are only meaningful for inputs that look like what it was trained on.

So we do three things, in this order:

1. Resize each 32x32 image to IMG_SIZE x IMG_SIZE. Upsampling adds no information, but it puts the content at the scale the filters expect.
2. Convert to a tensor, which also maps the pixel range to $[0, 1]$.
3. Normalise with the ImageNet mean and standard deviation, the ones printed in Task 1.

Resizing to 224 is the honest choice and it is the slow one. If it is too slow on your machine, 96 or 128 is a reasonable compromise: change IMG_SIZE and say so in your write-up.

Write your own description of what you did in the cell after the accuracy result. That description is a graded deliverable.

preprocess = transforms.Compose([
    transforms.Resize((IMG_SIZE, IMG_SIZE)),
    transforms.ToTensor(),
    transforms.Normalize(mean=IMAGENET_MEAN, std=IMAGENET_STD),
])

print(preprocess)

The data, and one split shared by every experiment

Task 4 compares three representations. The comparison only means something if all three see the same images, so we draw the subsample once, here, and reuse the indices everywhere.

dataset.data gives us the raw 32x32 uint8 arrays regardless of the transform, which is what the raw-pixel and orientation-histogram baselines need.

cifar_train = torchvision.datasets.CIFAR10(root=DATA_ROOT, train=True,
                                           download=True, transform=preprocess)
cifar_test = torchvision.datasets.CIFAR10(root=DATA_ROOT, train=False,
                                          download=True, transform=preprocess)

CLASS_NAMES = cifar_train.classes
print(CLASS_NAMES)

rng = np.random.default_rng(SEED)
train_idx = rng.choice(len(cifar_train), size=N_TRAIN, replace=False).tolist()
test_idx = rng.choice(len(cifar_test), size=N_TEST, replace=False).tolist()

y_train = np.array(cifar_train.targets)[train_idx]
y_test = np.array(cifar_test.targets)[test_idx]

raw_train = cifar_train.data[train_idx]     # (N_TRAIN, 32, 32, 3), uint8
raw_test = cifar_test.data[test_idx]        # (N_TEST, 32, 32, 3), uint8

print('train subset:', raw_train.shape, ' test subset:', raw_test.shape)
print('class counts in the training subset:', np.bincount(y_train))

# A look at the data we are working with.
fig, axes = plt.subplots(2, 6, figsize=(10, 3.6))
for ax, img, label in zip(axes.ravel(), raw_train, y_train):
    ax.imshow(img)
    ax.set_title(CLASS_NAMES[label], fontsize=9)
    ax.axis('off')
fig.suptitle('CIFAR10 at its native 32x32')
plt.tight_layout()
plt.show()

Extracting the frozen features once

The backbone never changes, and we use no data augmentation, so every image produces the same 512-dimensional vector in every epoch. Running the backbone once and caching the result is therefore identical to running it every epoch, and it is many times faster. Training the head then costs seconds.

nn.Sequential(*list(model.children())[:-1]) is the whole network except fc, which is exactly the penultimate representation described in Task 1.

This cell is the slow one. Expect a few minutes on a CPU.

backbone = nn.Sequential(*list(model.children())[:-1]).to(device).eval()


@torch.no_grad()
def extract_features(dataset, indices):
    """Run the frozen backbone over dataset[indices].

    Returns (features, labels) as CPU tensors of shape (n, 512) and (n,).
    """
    loader = DataLoader(Subset(dataset, indices), batch_size=BATCH_SIZE,
                        shuffle=False, num_workers=NUM_WORKERS)
    feats, labels = [], []
    for images, targets in loader:
        out = backbone(images.to(device)).flatten(1)
        feats.append(out.cpu())
        labels.append(targets)
    return torch.cat(feats), torch.cat(labels)


t0 = time.perf_counter()
F_train, L_train = extract_features(cifar_train, train_idx)
F_test, L_test = extract_features(cifar_test, test_idx)
feature_time = time.perf_counter() - t0

print(f'features: {tuple(F_train.shape)} train, {tuple(F_test.shape)} test')
print(f'extraction took {feature_time:.1f} s')

# The cached labels must agree with the ones we pulled out by index.
assert np.array_equal(L_train.numpy(), y_train)
assert np.array_equal(L_test.numpy(), y_test)

Training the head

Plain minibatch training of one linear layer, with cross-entropy loss. Only model.fc.parameters() goes to the optimiser, which is a second guard against accidentally training the backbone.

head = model.fc
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(head.parameters(), lr=LR)

Xtr = F_train.to(device)
ytr = torch.tensor(y_train, dtype=torch.long, device=device)
Xte = F_test.to(device)
yte = torch.tensor(y_test, dtype=torch.long, device=device)

history = {'loss': [], 'train_acc': [], 'test_acc': []}

t0 = time.perf_counter()
for epoch in range(EPOCHS):
    head.train()
    perm = torch.randperm(len(Xtr), device=device)
    running = 0.0
    for start in range(0, len(perm), BATCH_SIZE):
        batch = perm[start:start + BATCH_SIZE]
        optimizer.zero_grad()
        logits = head(Xtr[batch])
        loss = criterion(logits, ytr[batch])
        loss.backward()
        optimizer.step()
        running += loss.item() * len(batch)

    head.eval()
    with torch.no_grad():
        train_acc = (head(Xtr).argmax(1) == ytr).float().mean().item()
        test_acc = (head(Xte).argmax(1) == yte).float().mean().item()

    history['loss'].append(running / len(perm))
    history['train_acc'].append(train_acc)
    history['test_acc'].append(test_acc)
    print(f'epoch {epoch + 1:2d}  loss {history["loss"][-1]:.4f}  '
          f'train acc {train_acc:.3f}  test acc {test_acc:.3f}')

head_train_time = time.perf_counter() - t0
resnet_test_acc = history['test_acc'][-1]

print()
print(f'head trained in {head_train_time:.1f} s '
      f'(plus {feature_time:.1f} s of feature extraction)')
print(f'ResNet-18 + new head, test accuracy: {resnet_test_acc:.1%}')

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))
ax1.plot(range(1, EPOCHS + 1), history['loss'])
ax1.set_xlabel('epoch')
ax1.set_ylabel('training loss')
ax2.plot(range(1, EPOCHS + 1), history['train_acc'], label='train')
ax2.plot(range(1, EPOCHS + 1), history['test_acc'], label='test')
ax2.set_xlabel('epoch')
ax2.set_ylabel('accuracy')
ax2.set_ylim(0, 1)
ax2.legend()
plt.tight_layout()
plt.show()

A check worth running

The head was trained on cached features. The claim is that this is the same thing as training it inside the full network. Verify it: push one batch of test images through the complete model, from pixels to logits, and confirm the predictions match.

model.eval()
check_loader = DataLoader(Subset(cifar_test, test_idx[:BATCH_SIZE]),
                          batch_size=BATCH_SIZE, shuffle=False,
                          num_workers=NUM_WORKERS)
images, targets = next(iter(check_loader))

with torch.no_grad():
    end_to_end = model(images.to(device)).argmax(1).cpu()
    from_cache = head(F_test[:len(targets)].to(device)).argmax(1).cpu()

print('end-to-end and cached predictions agree:',
      torch.equal(end_to_end, from_cache))
print('accuracy on this batch:', (end_to_end == targets).float().mean().item())

Your description of the preprocessing

TODO. In a sentence or three, say what you did about input size and normalisation, what values you used, and why. If you changed IMG_SIZE, say what you changed it to and what it cost you. This is a graded deliverable.

Write your answer here.

Task 4: compare against baselines you train yourself

An accuracy figure on its own means nothing. Here we train two cheap baselines on the same images, the same split, and the same classifier, changing only the representation:

1. Raw pixels. The flattened 32x32x3 image, 3072 numbers.
2. A hand-crafted feature. The gradient orientation histogram you built in Lab 4.
3. Pretrained CNN features. The 512 numbers we cached above.

Every row uses scikit-learn's LogisticRegression, which is the same 10-way linear classifier in all three cases. Hold that fact: it is what the last question in this task is about.

def fit_linear_classifier(name, feature_name, Xtr, ytr, Xte, yte):
    """Train one 10-way logistic regression and report a table row."""
    t0 = time.perf_counter()
    clf = LogisticRegression(max_iter=MAX_ITER)
    clf.fit(Xtr, ytr)
    train_time = time.perf_counter() - t0
    acc = clf.score(Xte, yte)
    n_params = Xtr.shape[1] * NUM_CLASSES + NUM_CLASSES
    print(f'{feature_name:24s} dim {Xtr.shape[1]:5d}  '
          f'test acc {acc:.1%}  {train_time:.1f} s')
    return {'Model': name, 'Feature': feature_name,
            'Trainable parameters': n_params,
            'Test accuracy': f'{acc:.1%}',
            'Training time (s)': round(train_time, 1)}


rows = []

Row 1: raw pixels

Flatten, scale to $[0, 1]$, fit. Nothing else. A ConvergenceWarning here is not a problem; the classifier is simply still improving slowly when it hits MAX_ITER.

Xtr_raw = raw_train.reshape(len(raw_train), -1).astype(np.float32) / 255.0
Xte_raw = raw_test.reshape(len(raw_test), -1).astype(np.float32) / 255.0

rows.append(fit_linear_classifier('Linear', 'raw pixels',
                                  Xtr_raw, y_train, Xte_raw, y_test))

Row 2: your hand-crafted feature

TODO. Paste in your Lab 4 gradient orientation histogram.

The function below must take one CIFAR10 image, a uint8 array of shape (32, 32, 3), and return a one-dimensional feature vector of a fixed length. The Lab 4 recipe is: convert to grayscale, compute $I_x$ and $I_y$, form the gradient magnitude and orientation, then histogram the orientation with each pixel weighted by its magnitude. Normalise the histogram so that it sums to one, so that bright and dark images are comparable.

Lab 4 used 360 bins on a full-size KITTI image. CIFAR10 images are 32x32, so they hold about a thousand pixels and 360 bins would be almost empty. Use far fewer. N_ORIENT_BINS below is a starting point; try a couple of values and report what you chose.

Until you implement it, the cell after it will tell you the row is missing and the rest of the notebook will still run.

N_ORIENT_BINS = 36


def orientation_histogram(img_rgb, n_bins=N_ORIENT_BINS):
    """Return a 1-D feature vector for one CIFAR10 image.

    img_rgb : uint8 array of shape (32, 32, 3)
    returns : float array of shape (n_bins,)

    TODO: your Lab 4 construction goes here.
    """
    raise NotImplementedError('TODO: paste your Lab 4 orientation histogram.')


def orientation_features(images):
    """Apply orientation_histogram to a stack of images."""
    return np.stack([orientation_histogram(img) for img in images])

try:
    Xtr_hog = orientation_features(raw_train)
    Xte_hog = orientation_features(raw_test)
    rows.append(fit_linear_classifier('Linear', 'orientation histogram',
                                      Xtr_hog, y_train, Xte_hog, y_test))
except NotImplementedError as err:
    print('orientation histogram row skipped:', err)

Row 3: pretrained CNN features

The same LogisticRegression, on the 512-dimensional vectors the frozen backbone produced. This row and the head you trained in Task 3 are two ways of fitting a linear classifier to the same features, so their accuracies should land within a point or two of each other.

Xtr_cnn = F_train.numpy()
Xte_cnn = F_test.numpy()

rows.append(fit_linear_classifier('ResNet + new head', 'pretrained CNN',
                                  Xtr_cnn, y_train, Xte_cnn, y_test))

print()
print(f'for reference, the head trained in Task 3 scored {resnet_test_acc:.1%}')

The comparison table

This table is a graded deliverable. Note that the CNN row's training time excludes feature extraction; that cost is reported separately and you should mention it.

table = pd.DataFrame(rows, columns=['Model', 'Feature', 'Trainable parameters',
                                    'Test accuracy', 'Training time (s)'])
print(f'split: {N_TRAIN} training images, {N_TEST} test images, '
      f'input {IMG_SIZE}x{IMG_SIZE}')
print(f'CNN feature extraction (once, not in the table): {feature_time:.1f} s')
print()
print(table.to_string(index=False))
table

Answer these three questions

The handout asks for all three. Answer them in this cell.

1. Does the hand-crafted feature beat raw pixels? By how much?

Your answer here.

2. The pretrained ResNet never saw CIFAR10 during pretraining, and you retrained only the final layer. Why does it beat a feature that a human designed specifically for images?

Your answer here.

3. All three rows use the same linear classifier. So what exactly is the difference between them?

Your answer here.

Task 5: look at the features

Question 3 above claims that the representation is the only thing that changed. This task lets you see it.

Take a few hundred test images. Project their 512-dimensional CNN features down to two dimensions with PCA, and plot them coloured by true class. Then do exactly the same with the raw pixels of those same images. Two panels, side by side, same treatment.

PCA finds the two directions of largest variance and projects onto them. It is linear, so it cannot invent structure that the representation does not already hold. That is the point: whatever separation you see was there in the features.

pca_idx = np.arange(min(N_PCA, len(y_test)))
labels_pca = y_test[pca_idx]

cnn_2d = PCA(n_components=2, random_state=SEED).fit_transform(Xte_cnn[pca_idx])
raw_2d = PCA(n_components=2, random_state=SEED).fit_transform(Xte_raw[pca_idx])

fig, axes = plt.subplots(1, 2, figsize=(13, 5.5), constrained_layout=True)
cmap = plt.cm.tab10

for ax, points, title in [(axes[0], cnn_2d, 'Pretrained CNN features (512-d)'),
                          (axes[1], raw_2d, 'Raw pixels (3072-d)')]:
    for k in range(NUM_CLASSES):
        sel = labels_pca == k
        ax.scatter(points[sel, 0], points[sel, 1], s=12, alpha=0.7,
                   color=cmap(k), label=CLASS_NAMES[k])
    ax.set_title(f'{title}, PCA to 2D')
    ax.set_xlabel('PC 1')
    ax.set_ylabel('PC 2')

axes[1].legend(loc='center left', bbox_to_anchor=(1.02, 0.5), fontsize=9)
fig.suptitle(f'{len(pca_idx)} CIFAR10 test images, coloured by true class')
plt.show()

Discuss the two plots

This is the most persuasive figure in the lab, and the discussion is graded.

• Do the classes separate in the CNN panel? Which classes sit together, and does that grouping make sense?
• What does the raw-pixel panel look like? If it shows any structure at all, what is driving it? Brightness is a good thing to check.
• Both panels show the same images and the same projection method. Connect what you see here to the accuracy numbers in Task 4.

Your answer here.

Task 6 (optional): fine-tune the last residual block

This task is optional. The project report is due the same week. Skip it without penalty.

Unfreeze layer4, the last residual block, and train it together with the head. Every step now backpropagates through part of the backbone, and the features change every epoch, so the cached-feature trick no longer applies: each epoch must run the full network over every image. Expect it to be slow on a CPU.

Set RUN_TASK6 = True to try it, and use a small N_FT and one or two epochs. Report the accuracy change and the time change together; one without the other says nothing.

RUN_TASK6 = False
N_FT = 500        # images per fine-tuning epoch, keep this small on a CPU
FT_EPOCHS = 1
FT_LR = 1e-4

if RUN_TASK6:
    for param in model.layer4.parameters():
        param.requires_grad = True

    ft_trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print(f'trainable parameters now: {ft_trainable:,}')

    ft_params = list(model.layer4.parameters()) + list(model.fc.parameters())
    ft_optimizer = torch.optim.Adam(ft_params, lr=FT_LR)
    ft_loader = DataLoader(Subset(cifar_train, train_idx[:N_FT]),
                           batch_size=BATCH_SIZE, shuffle=True,
                           num_workers=NUM_WORKERS)

    t0 = time.perf_counter()
    model.train()
    for epoch in range(FT_EPOCHS):
        for images, targets in ft_loader:
            images, targets = images.to(device), targets.to(device)
            ft_optimizer.zero_grad()
            loss = criterion(model(images), targets)
            loss.backward()
            ft_optimizer.step()
        print(f'fine-tune epoch {epoch + 1}: last batch loss {loss.item():.4f}')
    ft_time = time.perf_counter() - t0

    model.eval()
    correct = 0
    test_loader = DataLoader(Subset(cifar_test, test_idx), batch_size=BATCH_SIZE,
                             shuffle=False, num_workers=NUM_WORKERS)
    with torch.no_grad():
        for images, targets in test_loader:
            preds = model(images.to(device)).argmax(1).cpu()
            correct += (preds == targets).sum().item()

    print(f'fine-tuned test accuracy : {correct / len(test_idx):.1%}')
    print(f'frozen-head accuracy was : {resnet_test_acc:.1%}')
    print(f'fine-tuning took {ft_time:.1f} s for {FT_EPOCHS} epoch(s) '
          f'over {N_FT} images')
else:
    print('Task 6 skipped.  Set RUN_TASK6 = True to run it.')

Before you submit

The handout's Deliverables section is the authority. Check your executed notebook against it:

• Pretrained model loaded, head replaced, trainable and frozen parameter counts printed (Task 2).
• Your description of how you handled resizing and normalisation (Task 3).
• Test accuracy for the ResNet with the new head (Task 3).
• The three-row comparison table, with all three baselines actually trained on the same split (Task 4).
• Answers to the three questions, including what differs between the rows given that the classifier is identical (Task 4).
• The two PCA plots side by side, both coloured by class, with your discussion (Task 5).

Submit through Canvas as a single executed notebook, run top to bottom, so that every figure and number above is visible in the file. Code that has not been executed cannot be marked.

Jupyter notebook

Source notebook is available here.

Compute resources

Everything above runs on a CPU. If you want a GPU anyway, the free tier of Google Colab is enough, and the Faculty of Science offers (experimentally) https://hubdev.science.ontariotechu.ca/.