{
"cells": [
{
"cell_type": "markdown",
"id": "cd3a98b4",
"metadata": {},
"source": [
"# Lung Cancer Detection from Chest CT Scans\n",
"\n",
"**A transfer-learning image classifier that distinguishes lung cancer subtypes from chest CT slices.**\n",
"\n",
"**Authors:** Azizbek Oqbutayev · Nozima Sotiboldiyeva\n",
"**Date:** 2026-06-16\n",
"**Tooling:** Python · PyTorch · scikit-learn · ONNX, built and documented with Claude Code (Opus 4.8)\n",
"\n",
"---\n",
"\n",
"This notebook documents the **entire** pipeline, step by step:\n",
"\n",
"1. Background — and a myth-busting note about the \"MIT lung cancer dataset\"\n",
"2. Environment & reproducibility\n",
"3. Loading the data and normalizing class labels\n",
"4. Exploratory data analysis (EDA)\n",
"5. Preprocessing & data augmentation\n",
"6. Model architecture (transfer learning with MobileNetV2)\n",
"7. Training\n",
"8. Evaluation (accuracy, per-class metrics, confusion matrix, ROC/AUC, clinical sensitivity/specificity)\n",
"9. Explainability with Grad-CAM\n",
"10. Exporting the model to ONNX for an in-browser demo\n",
"11. Results summary, limitations & references\n",
"\n",
"Every figure produced here is also saved into `../website/assets/` so the companion\n",
"website can render the same results visually."
]
},
{
"cell_type": "markdown",
"id": "1366f7e9",
"metadata": {},
"source": [
"## 1. Background — and the \"MIT dataset\" myth\n",
"\n",
"This project started from a research note investigating a claim about a *\"free lung cancer\n",
"dataset by MIT.\"* The finding was clear:\n",
"\n",
"> **There is no lung cancer detection dataset created by MIT the institution.** The phrase\n",
"> almost always comes from confusing **MIT the university** with the **MIT License** — a\n",
"> permissive open-source license attached to countless community datasets and repos on\n",
"> Kaggle, Hugging Face and GitHub. (MIT's actual medical-data lab maintains the MIMIC ICU /\n",
"> chest-X-ray collections via PhysioNet — credentialed-access data, *not* lung cancer sets.)\n",
"\n",
"The genuinely authoritative free lung-cancer datasets come from the **NIH/NCI** and **TCIA**\n",
"(LIDC-IDRI, LUNA16, NLST, NSCLC-Radiomics) plus academic sets like **LC25000**.\n",
"\n",
"The literal \"first dataset\" in that note is **LIDC-IDRI** — ~125 GB of 3-D CT volumes\n",
"requiring the TCIA downloader, DICOM handling and 3-D CNNs on a GPU. That is impractical to\n",
"download/train here and impossible to embed in a lightweight web demo. So we use a smaller,\n",
"fully imaging-based, citable set instead:\n",
"\n",
"**Dataset used: the Chest CT-Scan images set** (originally Kaggle / `mohamedhanyyy`, cited in\n",
"the source note's §B), mirrored on Hugging Face as `dorsar/lung-cancer`. Real chest-CT slices,\n",
"4 classes, pre-split into train/valid/test. Its license is — fittingly — **MIT**, which is a\n",
"live, real-world example of exactly the License-vs-institution confusion above."
]
},
{
"cell_type": "markdown",
"id": "53949d33",
"metadata": {},
"source": [
"## 2. Environment & reproducibility\n",
"\n",
"We fix random seeds for reproducibility and select the Apple-Silicon **MPS** GPU backend if\n",
"available (this notebook was developed on a MacBook Air M4), otherwise CPU."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "41dd1f9b",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-16T06:28:53.736062Z",
"iopub.status.busy": "2026-06-16T06:28:53.735685Z",
"iopub.status.idle": "2026-06-16T06:28:56.204861Z",
"shell.execute_reply": "2026-06-16T06:28:56.204426Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"PyTorch : 2.12.0\n",
"Torchvision : 0.27.0\n",
"Compute device: mps\n"
]
}
],
"source": [
"import os, random, json, time, re\n",
"from pathlib import Path\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns\n",
"from PIL import Image\n",
"\n",
"import torch\n",
"import torch.nn as nn\n",
"from torch.utils.data import Dataset, DataLoader\n",
"import torchvision\n",
"from torchvision import transforms\n",
"from torchvision.models import mobilenet_v2, MobileNet_V2_Weights\n",
"\n",
"SEED = 42\n",
"random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)\n",
"\n",
"if torch.backends.mps.is_available():\n",
" DEVICE = torch.device(\"mps\")\n",
"elif torch.cuda.is_available():\n",
" DEVICE = torch.device(\"cuda\")\n",
"else:\n",
" DEVICE = torch.device(\"cpu\")\n",
"\n",
"sns.set_theme(style=\"whitegrid\")\n",
"plt.rcParams[\"figure.dpi\"] = 110\n",
"\n",
"print(\"PyTorch :\", torch.__version__)\n",
"print(\"Torchvision :\", torchvision.__version__)\n",
"print(\"Compute device:\", DEVICE)"
]
},
{
"cell_type": "markdown",
"id": "f4e469c8",
"metadata": {},
"source": [
"### Project paths\n",
"\n",
"The notebook lives in `notebooks/`. We resolve the project root robustly and create the\n",
"output folders used by the website."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "03b954a2",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-16T06:28:56.206081Z",
"iopub.status.busy": "2026-06-16T06:28:56.205953Z",
"iopub.status.idle": "2026-06-16T06:28:56.208921Z",
"shell.execute_reply": "2026-06-16T06:28:56.208616Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Project root: /Users/abdulhamidnuriddinov/Documents/Projects/Video Editing with Claude/lung-cancer-detection\n",
"Data dir : /Users/abdulhamidnuriddinov/Documents/Projects/Video Editing with Claude/lung-cancer-detection/data/chest-ct/Data\n"
]
}
],
"source": [
"# Resolve project root whether the notebook runs from notebooks/ or the project root.\n",
"CANDIDATES = [\n",
" Path.cwd(), Path.cwd().parent, Path.cwd().parent.parent,\n",
" Path(\"/Users/abdulhamidnuriddinov/Documents/Projects/Video Editing with Claude/lung-cancer-detection\"),\n",
"]\n",
"ROOT = next((p for p in CANDIDATES if (p / \"data\" / \"chest-ct\" / \"Data\").exists()), Path.cwd().parent)\n",
"\n",
"DATA_DIR = ROOT / \"data\" / \"chest-ct\" / \"Data\"\n",
"MODELS_DIR = ROOT / \"models\"\n",
"WEB_DIR = ROOT / \"website\"\n",
"ASSETS_DIR = WEB_DIR / \"assets\"\n",
"for d in (MODELS_DIR, ASSETS_DIR):\n",
" d.mkdir(parents=True, exist_ok=True)\n",
"\n",
"assert DATA_DIR.exists(), f\"Dataset not found at {DATA_DIR}\"\n",
"print(\"Project root:\", ROOT)\n",
"print(\"Data dir :\", DATA_DIR)\n",
"\n",
"def savefig(name, **kw):\n",
" # Save the current matplotlib figure into the website assets folder.\n",
" plt.savefig(ASSETS_DIR / name, dpi=110, bbox_inches=\"tight\", **kw)"
]
},
{
"cell_type": "markdown",
"id": "4a407159",
"metadata": {},
"source": [
"## 3. Loading the data & normalizing class labels\n",
"\n",
"The dataset has four classes. In the **train/valid** splits the class folders carry extra\n",
"TNM-staging text (e.g. `adenocarcinoma_left.lower.lobe_T2_N0_M0_Ib`), while the **test** split\n",
"uses clean names. We normalize every folder to one of four canonical labels."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "c17316f5",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-16T06:28:56.209945Z",
"iopub.status.busy": "2026-06-16T06:28:56.209889Z",
"iopub.status.idle": "2026-06-16T06:28:56.214431Z",
"shell.execute_reply": "2026-06-16T06:28:56.214168Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train=400 valid=72 test=295\n"
]
}
],
"source": [
"CLASSES = [\"adenocarcinoma\", \"large.cell.carcinoma\", \"normal\", \"squamous.cell.carcinoma\"]\n",
"DISPLAY = {\n",
" \"adenocarcinoma\": \"Adenocarcinoma\",\n",
" \"large.cell.carcinoma\": \"Large cell carcinoma\",\n",
" \"normal\": \"Normal\",\n",
" \"squamous.cell.carcinoma\": \"Squamous cell carcinoma\",\n",
"}\n",
"CANCER_CLASSES = [\"adenocarcinoma\", \"large.cell.carcinoma\", \"squamous.cell.carcinoma\"]\n",
"\n",
"def folder_to_class(name):\n",
" n = name.lower()\n",
" if n.startswith(\"adenocarcinoma\"): return \"adenocarcinoma\"\n",
" if n.startswith(\"large\"): return \"large.cell.carcinoma\"\n",
" if n.startswith(\"squamous\"): return \"squamous.cell.carcinoma\"\n",
" if n.startswith(\"normal\"): return \"normal\"\n",
" raise ValueError(f\"Unrecognized class folder: {name}\")\n",
"\n",
"IMG_EXT = (\".png\", \".jpg\", \".jpeg\")\n",
"\n",
"def scan_split(split):\n",
" # Return list of (image_path, class_index) for a split.\n",
" split_dir = DATA_DIR / split\n",
" samples = []\n",
" for sub in sorted(os.listdir(split_dir)):\n",
" p = split_dir / sub\n",
" if not p.is_dir():\n",
" continue\n",
" label = folder_to_class(sub)\n",
" for f in sorted(os.listdir(p)):\n",
" if f.lower().endswith(IMG_EXT):\n",
" samples.append((str(p / f), CLASSES.index(label)))\n",
" return samples\n",
"\n",
"train_samples = scan_split(\"train\")\n",
"valid_samples = scan_split(\"valid\")\n",
"test_samples = scan_split(\"test\")\n",
"print(f\"train={len(train_samples)} valid={len(valid_samples)} test={len(test_samples)}\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "e04fbd44",
"metadata": {
"execution": {
"iopub.execute_input": "2026-06-16T06:28:56.215438Z",
"iopub.status.busy": "2026-06-16T06:28:56.215368Z",
"iopub.status.idle": "2026-06-16T06:28:56.226157Z",
"shell.execute_reply": "2026-06-16T06:28:56.225924Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" train | \n",
" valid | \n",
" test | \n",
" total | \n",
"
\n",
" \n",
" \n",
" \n",
" | Adenocarcinoma | \n",
" 100 | \n",
" 23 | \n",
" 100 | \n",
" 223 | \n",
"
\n",
" \n",
" | Large cell carcinoma | \n",
" 100 | \n",
" 21 | \n",
" 51 | \n",
" 172 | \n",
"
\n",
" \n",
" | Normal | \n",
" 100 | \n",
" 13 | \n",
" 54 | \n",
" 167 | \n",
"
\n",
" \n",
" | Squamous cell carcinoma | \n",
" 100 | \n",
" 15 | \n",
" 90 | \n",
" 205 | \n",
"
\n",
" \n",
" | TOTAL | \n",
" 400 | \n",
" 72 | \n",
" 295 | \n",
" 767 | \n",
"
\n",
" \n",
"
\n",
"