![](\"../ontario-tech-univ-logo.png\"){kind=logo}
\n",
"\n",
"![\"Creative](\"https://i.creativecommons.org/l/by-nc/4.0/88x31.png\")
\n", "The handout, not this notebook, lists what you must submit. See the\n", "Deliverables section of lab10. Submit a single\n", "notebook that has been executed top to bottom.\n", "
\n", "\n", "We take a ResNet trained on ImageNet, throw away its 1000-way classifier, bolt\n", "a 10-way classifier onto the frozen backbone, and train only that. Then we do\n", "the thing that gives the lab its point: we train the *same* linear classifier on\n", "raw pixels, on a feature you designed by hand in Lab 4, and on the pretrained\n", "network's features, and compare all three.\n", "\n", "**You do not need a GPU.** The backbone stays frozen, so the only thing being\n", "trained is one linear layer.\n", "\n", "**We use a subsample of CIFAR10.** Running a ResNet over all 60,000 images at\n", "224x224 on a CPU takes far too long for a lab session, so we draw a random\n", "subset. The sizes are the variables `N_TRAIN` and `N_TEST` in the setup cell.\n", "Raise them if you have time or a GPU, and say in your write-up what you used." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Setup\n", "\n", "Everything this notebook needs is in the course environment (see Lab 1). No\n", "`pip install` is required." ] }, { "cell_type": "code", "metadata": {}, "source": [ "import time\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "import torch\n", "import torch.nn as nn\n", "import torchvision\n", "from torch.utils.data import DataLoader, Subset\n", "from torchvision import models, transforms\n", "\n", "from sklearn.decomposition import PCA\n", "from sklearn.linear_model import LogisticRegression\n", "\n", "%matplotlib inline" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "# ==== configuration. Change these, do not scatter magic numbers below. =======\n", "\n", "SEED = 42\n", "\n", "DATA_ROOT = './data' # CIFAR10 is downloaded here the first time you run\n", "\n", "IMG_SIZE = 224 # what the pretrained ResNet expects. See Task 3.\n", "N_TRAIN = 5000 # subsample of the 50,000 CIFAR10 training images\n", "N_TEST = 1000 # subsample of the 10,000 CIFAR10 test images\n", "N_PCA = 500 # how many test images to plot in Task 5\n", "\n", "BATCH_SIZE = 64\n", "NUM_WORKERS = 0 # set to 2 on Colab for faster loading\n", "EPOCHS = 15 # epochs for the linear head\n", "LR = 1e-3\n", "MAX_ITER = 500 # iterations for scikit-learn's logistic regression\n", "\n", "# ImageNet channel statistics. These are a property of the pretrained model,\n", "# not of CIFAR10. Do not replace them with CIFAR10's own mean and std.\n", "IMAGENET_MEAN = [0.485, 0.456, 0.406]\n", "IMAGENET_STD = [0.229, 0.224, 0.225]\n", "\n", "torch.manual_seed(SEED)\n", "np.random.seed(SEED)\n", "\n", "device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n", "print('device:', device)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 1: get a pretrained model\n", "\n", "We load ResNet-18 with its ImageNet weights. ResNet-18 is the smallest of the\n", "family and is plenty for this lab.\n", "\n", "The handout points you at [`timm`](https://timm.fast.ai/), which is an excellent\n", "library and worth knowing. We use `torchvision` here because it ships with\n", "PyTorch and needs no extra install. The `timm` equivalent is one line:\n", "\n", "~~~python\n", "import timm\n", "resnet = timm.create_model('resnet18', pretrained=True)\n", "~~~\n", "\n", "Either route is acceptable. Everything below assumes the `torchvision` model." ] }, { "cell_type": "code", "metadata": {}, "source": [ "weights = models.ResNet18_Weights.IMAGENET1K_V1\n", "resnet = models.resnet18(weights=weights)\n", "resnet.eval()\n", "\n", "print(resnet)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Scroll to the bottom of that printout. The last two lines are the interesting\n", "ones:\n", "\n", "- `avgpool`, which collapses each of the 512 final feature maps to one number.\n", " Its output is the **penultimate representation**, a 512-dimensional vector per\n", " image. We use it again in Tasks 4 and 5.\n", "- `fc`, a `Linear(512, 1000)`. That is the **classifier head**, and it is the\n", " part we replace.\n", "\n", "The weights object also records how the model was trained. Read the input size\n", "and the normalisation off it rather than guessing." ] }, { "cell_type": "code", "metadata": {}, "source": [ "print('classifier head :', resnet.fc)\n", "print('feature size :', resnet.fc.in_features)\n", "print()\n", "print('preprocessing the weights were trained with:')\n", "print(weights.transforms())" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 2: swap the head and freeze the backbone\n", "\n", "Two steps, in this order.\n", "\n", "1. Set `requires_grad = False` on every parameter, which freezes the whole\n", " network.\n", "2. Replace `fc` with a fresh `Linear(512, 10)`. A newly created layer has\n", " `requires_grad = True`, so the head trains and nothing else does.\n", "\n", "Do it the other way round and you freeze your own head as well.\n", "\n", "The parameter count is the check that tells you it worked. A `Linear(512, 10)`\n", "holds $512 \\times 10$ weights plus 10 biases, which is $5130$ numbers. If the\n", "printout below says anything else, the freeze went wrong. Compare against the\n", "frozen count: roughly 11 million. That ratio is why this lab runs on a laptop." ] }, { "cell_type": "code", "metadata": {}, "source": [ "NUM_CLASSES = 10\n", "\n", "num_features = resnet.fc.in_features # 512 for ResNet-18\n", "\n", "for param in resnet.parameters(): # 1. freeze everything\n", " param.requires_grad = False\n", "\n", "resnet.fc = nn.Linear(num_features, NUM_CLASSES) # 2. fresh, trainable head\n", "\n", "model = resnet.to(device)\n", "\n", "trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)\n", "frozen = sum(p.numel() for p in model.parameters() if not p.requires_grad)\n", "expected = num_features * NUM_CLASSES + NUM_CLASSES\n", "\n", "print(f'penultimate feature size : {num_features}')\n", "print(f'trainable parameters : {trainable:,}')\n", "print(f'frozen parameters : {frozen:,}')\n", "print(f'trainable fraction : {trainable / (trainable + frozen):.5%}')\n", "print()\n", "print(f'head should hold {num_features} x {NUM_CLASSES} weights + {NUM_CLASSES} '\n", " f'biases = {expected:,}')\n", "\n", "assert trainable == expected == 5130, 'the freeze or the head is wrong'" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 3: train the head and evaluate\n", "\n", "## Preprocessing, which is the part that catches people\n", "\n", "CIFAR10 images are 32x32, and they are stored as 8-bit RGB. The pretrained\n", "ResNet was trained on 224x224 crops of ImageNet, normalised with ImageNet's\n", "channel mean and standard deviation. Feed it 32x32 images, or normalise with\n", "CIFAR10's own statistics, and the accuracy collapses. The features the backbone\n", "computes are only meaningful for inputs that look like what it was trained on.\n", "\n", "So we do three things, in this order:\n", "\n", "1. **Resize** each 32x32 image to `IMG_SIZE` x `IMG_SIZE`. Upsampling adds no\n", " information, but it puts the content at the scale the filters expect.\n", "2. **Convert to a tensor**, which also maps the pixel range to $[0, 1]$.\n", "3. **Normalise** with the ImageNet mean and standard deviation, the ones printed\n", " in Task 1.\n", "\n", "Resizing to 224 is the honest choice and it is the slow one. If it is too slow\n", "on your machine, 96 or 128 is a reasonable compromise: change `IMG_SIZE` and say\n", "so in your write-up.\n", "\n", "Write your own description of what you did in the cell after the accuracy\n", "result. That description is a graded deliverable." ] }, { "cell_type": "code", "metadata": {}, "source": [ "preprocess = transforms.Compose([\n", " transforms.Resize((IMG_SIZE, IMG_SIZE)),\n", " transforms.ToTensor(),\n", " transforms.Normalize(mean=IMAGENET_MEAN, std=IMAGENET_STD),\n", "])\n", "\n", "print(preprocess)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The data, and one split shared by every experiment\n", "\n", "Task 4 compares three representations. The comparison only means something if\n", "all three see the same images, so we draw the subsample once, here, and reuse\n", "the indices everywhere.\n", "\n", "`dataset.data` gives us the raw 32x32 uint8 arrays regardless of the transform,\n", "which is what the raw-pixel and orientation-histogram baselines need." ] }, { "cell_type": "code", "metadata": {}, "source": [ "cifar_train = torchvision.datasets.CIFAR10(root=DATA_ROOT, train=True,\n", " download=True, transform=preprocess)\n", "cifar_test = torchvision.datasets.CIFAR10(root=DATA_ROOT, train=False,\n", " download=True, transform=preprocess)\n", "\n", "CLASS_NAMES = cifar_train.classes\n", "print(CLASS_NAMES)\n", "\n", "rng = np.random.default_rng(SEED)\n", "train_idx = rng.choice(len(cifar_train), size=N_TRAIN, replace=False).tolist()\n", "test_idx = rng.choice(len(cifar_test), size=N_TEST, replace=False).tolist()\n", "\n", "y_train = np.array(cifar_train.targets)[train_idx]\n", "y_test = np.array(cifar_test.targets)[test_idx]\n", "\n", "raw_train = cifar_train.data[train_idx] # (N_TRAIN, 32, 32, 3), uint8\n", "raw_test = cifar_test.data[test_idx] # (N_TEST, 32, 32, 3), uint8\n", "\n", "print('train subset:', raw_train.shape, ' test subset:', raw_test.shape)\n", "print('class counts in the training subset:', np.bincount(y_train))" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "# A look at the data we are working with.\n", "fig, axes = plt.subplots(2, 6, figsize=(10, 3.6))\n", "for ax, img, label in zip(axes.ravel(), raw_train, y_train):\n", " ax.imshow(img)\n", " ax.set_title(CLASS_NAMES[label], fontsize=9)\n", " ax.axis('off')\n", "fig.suptitle('CIFAR10 at its native 32x32')\n", "plt.tight_layout()\n", "plt.show()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Extracting the frozen features once\n", "\n", "The backbone never changes, and we use no data augmentation, so every image\n", "produces the same 512-dimensional vector in every epoch. Running the backbone\n", "once and caching the result is therefore *identical* to running it every epoch,\n", "and it is many times faster. Training the head then costs seconds.\n", "\n", "`nn.Sequential(*list(model.children())[:-1])` is the whole network except `fc`,\n", "which is exactly the penultimate representation described in Task 1.\n", "\n", "This cell is the slow one. Expect a few minutes on a CPU." ] }, { "cell_type": "code", "metadata": {}, "source": [ "backbone = nn.Sequential(*list(model.children())[:-1]).to(device).eval()\n", "\n", "\n", "@torch.no_grad()\n", "def extract_features(dataset, indices):\n", " \"\"\"Run the frozen backbone over dataset[indices].\n", "\n", " Returns (features, labels) as CPU tensors of shape (n, 512) and (n,).\n", " \"\"\"\n", " loader = DataLoader(Subset(dataset, indices), batch_size=BATCH_SIZE,\n", " shuffle=False, num_workers=NUM_WORKERS)\n", " feats, labels = [], []\n", " for images, targets in loader:\n", " out = backbone(images.to(device)).flatten(1)\n", " feats.append(out.cpu())\n", " labels.append(targets)\n", " return torch.cat(feats), torch.cat(labels)\n", "\n", "\n", "t0 = time.perf_counter()\n", "F_train, L_train = extract_features(cifar_train, train_idx)\n", "F_test, L_test = extract_features(cifar_test, test_idx)\n", "feature_time = time.perf_counter() - t0\n", "\n", "print(f'features: {tuple(F_train.shape)} train, {tuple(F_test.shape)} test')\n", "print(f'extraction took {feature_time:.1f} s')\n", "\n", "# The cached labels must agree with the ones we pulled out by index.\n", "assert np.array_equal(L_train.numpy(), y_train)\n", "assert np.array_equal(L_test.numpy(), y_test)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Training the head\n", "\n", "Plain minibatch training of one linear layer, with cross-entropy loss. Only\n", "`model.fc.parameters()` goes to the optimiser, which is a second guard against\n", "accidentally training the backbone." ] }, { "cell_type": "code", "metadata": {}, "source": [ "head = model.fc\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = torch.optim.Adam(head.parameters(), lr=LR)\n", "\n", "Xtr = F_train.to(device)\n", "ytr = torch.tensor(y_train, dtype=torch.long, device=device)\n", "Xte = F_test.to(device)\n", "yte = torch.tensor(y_test, dtype=torch.long, device=device)\n", "\n", "history = {'loss': [], 'train_acc': [], 'test_acc': []}\n", "\n", "t0 = time.perf_counter()\n", "for epoch in range(EPOCHS):\n", " head.train()\n", " perm = torch.randperm(len(Xtr), device=device)\n", " running = 0.0\n", " for start in range(0, len(perm), BATCH_SIZE):\n", " batch = perm[start:start + BATCH_SIZE]\n", " optimizer.zero_grad()\n", " logits = head(Xtr[batch])\n", " loss = criterion(logits, ytr[batch])\n", " loss.backward()\n", " optimizer.step()\n", " running += loss.item() * len(batch)\n", "\n", " head.eval()\n", " with torch.no_grad():\n", " train_acc = (head(Xtr).argmax(1) == ytr).float().mean().item()\n", " test_acc = (head(Xte).argmax(1) == yte).float().mean().item()\n", "\n", " history['loss'].append(running / len(perm))\n", " history['train_acc'].append(train_acc)\n", " history['test_acc'].append(test_acc)\n", " print(f'epoch {epoch + 1:2d} loss {history[\"loss\"][-1]:.4f} '\n", " f'train acc {train_acc:.3f} test acc {test_acc:.3f}')\n", "\n", "head_train_time = time.perf_counter() - t0\n", "resnet_test_acc = history['test_acc'][-1]\n", "\n", "print()\n", "print(f'head trained in {head_train_time:.1f} s '\n", " f'(plus {feature_time:.1f} s of feature extraction)')\n", "print(f'ResNet-18 + new head, test accuracy: {resnet_test_acc:.1%}')" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(10, 3.5))\n", "ax1.plot(range(1, EPOCHS + 1), history['loss'])\n", "ax1.set_xlabel('epoch')\n", "ax1.set_ylabel('training loss')\n", "ax2.plot(range(1, EPOCHS + 1), history['train_acc'], label='train')\n", "ax2.plot(range(1, EPOCHS + 1), history['test_acc'], label='test')\n", "ax2.set_xlabel('epoch')\n", "ax2.set_ylabel('accuracy')\n", "ax2.set_ylim(0, 1)\n", "ax2.legend()\n", "plt.tight_layout()\n", "plt.show()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## A check worth running\n", "\n", "The head was trained on cached features. The claim is that this is the same\n", "thing as training it inside the full network. Verify it: push one batch of test\n", "images through the complete `model`, from pixels to logits, and confirm the\n", "predictions match." ] }, { "cell_type": "code", "metadata": {}, "source": [ "model.eval()\n", "check_loader = DataLoader(Subset(cifar_test, test_idx[:BATCH_SIZE]),\n", " batch_size=BATCH_SIZE, shuffle=False,\n", " num_workers=NUM_WORKERS)\n", "images, targets = next(iter(check_loader))\n", "\n", "with torch.no_grad():\n", " end_to_end = model(images.to(device)).argmax(1).cpu()\n", " from_cache = head(F_test[:len(targets)].to(device)).argmax(1).cpu()\n", "\n", "print('end-to-end and cached predictions agree:',\n", " torch.equal(end_to_end, from_cache))\n", "print('accuracy on this batch:', (end_to_end == targets).float().mean().item())" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Your description of the preprocessing\n", "\n", "**TODO.** In a sentence or three, say what you did about input size and\n", "normalisation, what values you used, and why. If you changed `IMG_SIZE`, say\n", "what you changed it to and what it cost you. This is a graded deliverable.\n", "\n", "*Write your answer here.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 4: compare against baselines you train yourself\n", "\n", "An accuracy figure on its own means nothing. Here we train two cheap baselines\n", "on the same images, the same split, and the same classifier, changing only the\n", "representation:\n", "\n", "1. **Raw pixels.** The flattened 32x32x3 image, 3072 numbers.\n", "2. **A hand-crafted feature.** The gradient orientation histogram you built in\n", " Lab 4.\n", "3. **Pretrained CNN features.** The 512 numbers we cached above.\n", "\n", "Every row uses scikit-learn's `LogisticRegression`, which is the same 10-way\n", "linear classifier in all three cases. Hold that fact: it is what the last\n", "question in this task is about." ] }, { "cell_type": "code", "metadata": {}, "source": [ "def fit_linear_classifier(name, feature_name, Xtr, ytr, Xte, yte):\n", " \"\"\"Train one 10-way logistic regression and report a table row.\"\"\"\n", " t0 = time.perf_counter()\n", " clf = LogisticRegression(max_iter=MAX_ITER)\n", " clf.fit(Xtr, ytr)\n", " train_time = time.perf_counter() - t0\n", " acc = clf.score(Xte, yte)\n", " n_params = Xtr.shape[1] * NUM_CLASSES + NUM_CLASSES\n", " print(f'{feature_name:24s} dim {Xtr.shape[1]:5d} '\n", " f'test acc {acc:.1%} {train_time:.1f} s')\n", " return {'Model': name, 'Feature': feature_name,\n", " 'Trainable parameters': n_params,\n", " 'Test accuracy': f'{acc:.1%}',\n", " 'Training time (s)': round(train_time, 1)}\n", "\n", "\n", "rows = []" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Row 1: raw pixels\n", "\n", "Flatten, scale to $[0, 1]$, fit. Nothing else. A `ConvergenceWarning` here is\n", "not a problem; the classifier is simply still improving slowly when it hits\n", "`MAX_ITER`." ] }, { "cell_type": "code", "metadata": {}, "source": [ "Xtr_raw = raw_train.reshape(len(raw_train), -1).astype(np.float32) / 255.0\n", "Xte_raw = raw_test.reshape(len(raw_test), -1).astype(np.float32) / 255.0\n", "\n", "rows.append(fit_linear_classifier('Linear', 'raw pixels',\n", " Xtr_raw, y_train, Xte_raw, y_test))" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Row 2: your hand-crafted feature\n", "\n", "**TODO.** Paste in your Lab 4 gradient orientation histogram.\n", "\n", "The function below must take one CIFAR10 image, a `uint8` array of shape\n", "`(32, 32, 3)`, and return a one-dimensional feature vector of a fixed length.\n", "The Lab 4 recipe is: convert to grayscale, compute $I_x$ and $I_y$, form the\n", "gradient magnitude and orientation, then histogram the orientation with each\n", "pixel weighted by its magnitude. Normalise the histogram so that it sums to\n", "one, so that bright and dark images are comparable.\n", "\n", "Lab 4 used 360 bins on a full-size KITTI image. CIFAR10 images are 32x32, so\n", "they hold about a thousand pixels and 360 bins would be almost empty. Use far\n", "fewer. `N_ORIENT_BINS` below is a starting point; try a couple of values and\n", "report what you chose.\n", "\n", "Until you implement it, the cell after it will tell you the row is missing and\n", "the rest of the notebook will still run." ] }, { "cell_type": "code", "metadata": {}, "source": [ "N_ORIENT_BINS = 36\n", "\n", "\n", "def orientation_histogram(img_rgb, n_bins=N_ORIENT_BINS):\n", " \"\"\"Return a 1-D feature vector for one CIFAR10 image.\n", "\n", " img_rgb : uint8 array of shape (32, 32, 3)\n", " returns : float array of shape (n_bins,)\n", "\n", " TODO: your Lab 4 construction goes here.\n", " \"\"\"\n", " raise NotImplementedError('TODO: paste your Lab 4 orientation histogram.')\n", "\n", "\n", "def orientation_features(images):\n", " \"\"\"Apply orientation_histogram to a stack of images.\"\"\"\n", " return np.stack([orientation_histogram(img) for img in images])" ], "execution_count": null, "outputs": [] }, { "cell_type": "code", "metadata": {}, "source": [ "try:\n", " Xtr_hog = orientation_features(raw_train)\n", " Xte_hog = orientation_features(raw_test)\n", " rows.append(fit_linear_classifier('Linear', 'orientation histogram',\n", " Xtr_hog, y_train, Xte_hog, y_test))\n", "except NotImplementedError as err:\n", " print('orientation histogram row skipped:', err)" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Row 3: pretrained CNN features\n", "\n", "The same `LogisticRegression`, on the 512-dimensional vectors the frozen\n", "backbone produced. This row and the head you trained in Task 3 are two ways of\n", "fitting a linear classifier to the same features, so their accuracies should\n", "land within a point or two of each other." ] }, { "cell_type": "code", "metadata": {}, "source": [ "Xtr_cnn = F_train.numpy()\n", "Xte_cnn = F_test.numpy()\n", "\n", "rows.append(fit_linear_classifier('ResNet + new head', 'pretrained CNN',\n", " Xtr_cnn, y_train, Xte_cnn, y_test))\n", "\n", "print()\n", "print(f'for reference, the head trained in Task 3 scored {resnet_test_acc:.1%}')" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The comparison table\n", "\n", "This table is a graded deliverable. Note that the CNN row's training time\n", "excludes feature extraction; that cost is reported separately and you should\n", "mention it." ] }, { "cell_type": "code", "metadata": {}, "source": [ "table = pd.DataFrame(rows, columns=['Model', 'Feature', 'Trainable parameters',\n", " 'Test accuracy', 'Training time (s)'])\n", "print(f'split: {N_TRAIN} training images, {N_TEST} test images, '\n", " f'input {IMG_SIZE}x{IMG_SIZE}')\n", "print(f'CNN feature extraction (once, not in the table): {feature_time:.1f} s')\n", "print()\n", "print(table.to_string(index=False))\n", "table" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Answer these three questions\n", "\n", "The handout asks for all three. Answer them in this cell.\n", "\n", "**1. Does the hand-crafted feature beat raw pixels? By how much?**\n", "\n", "*Your answer here.*\n", "\n", "**2. The pretrained ResNet never saw CIFAR10 during pretraining, and you\n", "retrained only the final layer. Why does it beat a feature that a human\n", "designed specifically for images?**\n", "\n", "*Your answer here.*\n", "\n", "**3. All three rows use the same linear classifier. So what exactly is the\n", "difference between them?**\n", "\n", "*Your answer here.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 5: look at the features\n", "\n", "Question 3 above claims that the representation is the only thing that changed.\n", "This task lets you see it.\n", "\n", "Take a few hundred test images. Project their 512-dimensional CNN features down\n", "to two dimensions with PCA, and plot them coloured by true class. Then do\n", "exactly the same with the raw pixels of those same images. Two panels, side by\n", "side, same treatment.\n", "\n", "PCA finds the two directions of largest variance and projects onto them. It is\n", "linear, so it cannot invent structure that the representation does not already\n", "hold. That is the point: whatever separation you see was there in the features." ] }, { "cell_type": "code", "metadata": {}, "source": [ "pca_idx = np.arange(min(N_PCA, len(y_test)))\n", "labels_pca = y_test[pca_idx]\n", "\n", "cnn_2d = PCA(n_components=2, random_state=SEED).fit_transform(Xte_cnn[pca_idx])\n", "raw_2d = PCA(n_components=2, random_state=SEED).fit_transform(Xte_raw[pca_idx])\n", "\n", "fig, axes = plt.subplots(1, 2, figsize=(13, 5.5), constrained_layout=True)\n", "cmap = plt.cm.tab10\n", "\n", "for ax, points, title in [(axes[0], cnn_2d, 'Pretrained CNN features (512-d)'),\n", " (axes[1], raw_2d, 'Raw pixels (3072-d)')]:\n", " for k in range(NUM_CLASSES):\n", " sel = labels_pca == k\n", " ax.scatter(points[sel, 0], points[sel, 1], s=12, alpha=0.7,\n", " color=cmap(k), label=CLASS_NAMES[k])\n", " ax.set_title(f'{title}, PCA to 2D')\n", " ax.set_xlabel('PC 1')\n", " ax.set_ylabel('PC 2')\n", "\n", "axes[1].legend(loc='center left', bbox_to_anchor=(1.02, 0.5), fontsize=9)\n", "fig.suptitle(f'{len(pca_idx)} CIFAR10 test images, coloured by true class')\n", "plt.show()" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Discuss the two plots\n", "\n", "This is the most persuasive figure in the lab, and the discussion is graded.\n", "\n", "- Do the classes separate in the CNN panel? Which classes sit together, and\n", " does that grouping make sense?\n", "- What does the raw-pixel panel look like? If it shows any structure at all,\n", " what is driving it? Brightness is a good thing to check.\n", "- Both panels show the same images and the same projection method. Connect what\n", " you see here to the accuracy numbers in Task 4.\n", "\n", "*Your answer here.*" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Task 6 (optional): fine-tune the last residual block\n", "\n", "**This task is optional.** The project report is due the same week. Skip it\n", "without penalty.\n", "\n", "Unfreeze `layer4`, the last residual block, and train it together with the head.\n", "Every step now backpropagates through part of the backbone, and the features\n", "change every epoch, so the cached-feature trick no longer applies: each epoch\n", "must run the full network over every image. Expect it to be slow on a CPU.\n", "\n", "Set `RUN_TASK6 = True` to try it, and use a small `N_FT` and one or two epochs.\n", "Report the accuracy change and the time change together; one without the other\n", "says nothing." ] }, { "cell_type": "code", "metadata": {}, "source": [ "RUN_TASK6 = False\n", "N_FT = 500 # images per fine-tuning epoch, keep this small on a CPU\n", "FT_EPOCHS = 1\n", "FT_LR = 1e-4\n", "\n", "if RUN_TASK6:\n", " for param in model.layer4.parameters():\n", " param.requires_grad = True\n", "\n", " ft_trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)\n", " print(f'trainable parameters now: {ft_trainable:,}')\n", "\n", " ft_params = list(model.layer4.parameters()) + list(model.fc.parameters())\n", " ft_optimizer = torch.optim.Adam(ft_params, lr=FT_LR)\n", " ft_loader = DataLoader(Subset(cifar_train, train_idx[:N_FT]),\n", " batch_size=BATCH_SIZE, shuffle=True,\n", " num_workers=NUM_WORKERS)\n", "\n", " t0 = time.perf_counter()\n", " model.train()\n", " for epoch in range(FT_EPOCHS):\n", " for images, targets in ft_loader:\n", " images, targets = images.to(device), targets.to(device)\n", " ft_optimizer.zero_grad()\n", " loss = criterion(model(images), targets)\n", " loss.backward()\n", " ft_optimizer.step()\n", " print(f'fine-tune epoch {epoch + 1}: last batch loss {loss.item():.4f}')\n", " ft_time = time.perf_counter() - t0\n", "\n", " model.eval()\n", " correct = 0\n", " test_loader = DataLoader(Subset(cifar_test, test_idx), batch_size=BATCH_SIZE,\n", " shuffle=False, num_workers=NUM_WORKERS)\n", " with torch.no_grad():\n", " for images, targets in test_loader:\n", " preds = model(images.to(device)).argmax(1).cpu()\n", " correct += (preds == targets).sum().item()\n", "\n", " print(f'fine-tuned test accuracy : {correct / len(test_idx):.1%}')\n", " print(f'frozen-head accuracy was : {resnet_test_acc:.1%}')\n", " print(f'fine-tuning took {ft_time:.1f} s for {FT_EPOCHS} epoch(s) '\n", " f'over {N_FT} images')\n", "else:\n", " print('Task 6 skipped. Set RUN_TASK6 = True to run it.')" ], "execution_count": null, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Before you submit\n", "\n", "The handout's **Deliverables** section is the authority. Check your executed\n", "notebook against it:\n", "\n", "- [ ] Pretrained model loaded, head replaced, trainable and frozen parameter\n", " counts printed (Task 2).\n", "- [ ] Your description of how you handled resizing and normalisation (Task 3).\n", "- [ ] Test accuracy for the ResNet with the new head (Task 3).\n", "- [ ] The three-row comparison table, with all three baselines actually trained\n", " on the same split (Task 4).\n", "- [ ] Answers to the three questions, including what differs between the rows\n", " given that the classifier is identical (Task 4).\n", "- [ ] The two PCA plots side by side, both coloured by class, with your\n", " discussion (Task 5).\n", "\n", "Submit through Canvas as a **single executed notebook**, run top to bottom, so\n", "that every figure and number above is visible in the file. Code that has not\n", "been executed cannot be marked." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "