Project: multi-input models for OCR

Crumb buys from four regular suppliers: Deccan Dairy for butter and cream, Shree Flour Mills, Konkan Cocoa and Pune Packaging. Every delivery comes with a paper invoice. Every invoice gets typed into the accounts system by hand: invoice number, date, total amount and the batch codes of what was delivered. About 400 invoices a month, roughly six hours of typing a week.

The typing was tolerable. What changed Meera’s mind was the butter recall. In September, Deccan Dairy recalled one batch of salted butter, code SBO-4412, over a packaging fault. Crumb’s batch codes lived only on paper, in a box file, in the accounts office. It took two people two days to work out which stores had received that batch and which pastries it had gone into. Nobody got ill. Meera did not want to find out what happens next time.

“I want every batch code in the system the day it arrives,” she said. “Can the computer read the invoices?”

Arjun had used vision-capable LLMs in Module 12. “We could send photos to a multimodal model.”

“For messy, unpredictable documents, that’s often the right call,” Priya said. “But look at what we actually have. Four suppliers. Fixed invoice layouts, so cropping the fields is easy. Store staff photographing invoices on cheap tablets, sometimes with no signal in the back storeroom. Thousands of tiny crops a month. And one very specific pain: Deccan’s batch codes are three letters from B, O, S, I, Z and G, while Shree Flour’s are three digits from 8, 0, 5, 1, 2 and 6. Blur a phone photo a little and SBO and 580 look almost identical. A model that knows which supplier the invoice came from can tell them apart. That’s this module’s lesson in one project: two kinds of input, two kinds of output.”

The brief

OCR (optical character recognition) means reading text from images. A full invoice reader has two stages: find the fields on the page, then read each field. Because Crumb’s four suppliers use fixed layouts, the first stage can be a simple template per supplier: “the invoice number is always in this rectangle”. This project is the second stage, the hard part: given a small crop of one field, output its text.

Each training example has:

• An image: a greyscale crop, 32 pixels tall and 128 wide, of one printed field, from either a flatbed scanner (clean) or a phone camera (tilted, blurred, unevenly lit, noisy).
• Metadata: which supplier issued the invoice, and which device captured it.
• Labels: the exact text, and the field type (date, amount, invoice number or batch code).

Every supplier prints dates, amounts and invoice numbers its own way (14/10/2026 against 2026-10-14, Rs 1,245.50 against 1245.50), and batch codes follow per-supplier patterns of confusable characters. Real invoices are confidential, so you will generate realistic synthetic crops, which also gives you unlimited labelled data and full control over how hard the task is.

What you will build

Must have:

• A synthetic crop generator with supplier-specific formats and two capture devices.
• A Dataset and a CTC-friendly collate function.
• A CNN encoder, a metadata encoder fused at every step, a BiLSTM, a CTC text head and a field-type head.
• A weighted multi-head training loop that logs each loss separately.
• Greedy decoding and evaluation with character error rate and exact match, by device and field type.
• Ablations on the auxiliary weight and on metadata.

Nice to have:

• A “show me the hard cases” report of the worst-read crops.
• Format checks after decoding (a date must parse, an amount must be a number).
• A test on a few real photographed invoices.

Architecture

flowchart LR
    A["Field crop 1x32x128"] --> B["CNN encoder"]
    B --> C["32 steps x 128 features"]
    D["Supplier, device"] --> E["Embeddings + MLP"]
    E --> F["Copy to all 32 steps"]
    C --> G["Concatenate: 32 x 160"]
    F --> G
    G --> H["BiLSTM: 32 x 256"]
    H --> I["Text head + CTC loss"]
    H --> J["Mean over steps"]
    J --> K["Field-type head + cross-entropy"]

Field crop and CNN encoder. A 32×128 greyscale crop goes through four convolution blocks from Module 28. The last two pool only the height, so the crop ends up 1 pixel tall and 32 wide with 128 channels: a left-to-right sequence of 32 vectors, each describing a vertical slice of the text.

Metadata encoder. Supplier and device are categories, so each gets an embedding (a small learned vector per category, from this module’s multi-IO lesson): 8 numbers per supplier, 4 per device. A small MLP turns those 12 numbers into 32.

Fusion. The metadata vector is the same for the whole crop, but the image is a sequence. So the metadata is copied to all 32 steps and concatenated with each step’s image features: 128 + 32 = 160 numbers per step. Now every step “knows” which supplier it is reading.

BiLSTM. A bidirectional LSTM reads the 32 steps left to right and right to left, so each step’s output (128 + 128 = 256 numbers) knows about the characters on both sides. That context is what lets the model decide whether a blurry shape is S or 5.

Text head and CTC loss. At each step, a score for every character plus a special blank, trained with CTC loss (milestone 5).

Field-type head. The 32 steps are averaged into one vector and classified into the four field types. This is an auxiliary loss: Crumb barely needs the field type (the template already knows it), but predicting it might push the shared layers to learn the shape of dates, amounts and codes. Whether it actually helps is something you will measure, not assume.

Tech stack

Piece	Tool	Why
Framework	PyTorch	Multi-input models, CTC loss and BiLSTMs are all built in
Image generation	Pillow (PIL)	Draws text with a built-in scalable font, applies blur and rotation
Arrays	NumPy	Noise and lighting effects on the rendered crops
Metrics	torchmetrics CharErrorRate	Standard, tested character error rate
Hardware	Laptop CPU or GPU	About 10 to 25 seconds per epoch for 6,000 crops on an M-series Mac

uv init invoice-reader && cd invoice-reader
uv add torch torchvision torchmetrics pillow numpy

Pillow 10.1 or newer is required for ImageFont.load_default(size=...). Library APIs change quickly. If a name has moved, check the PyTorch tutorials and the torchmetrics documentation.

Build it, milestone by milestone

1. Generate a synthetic invoice-field dataset

Why. You need thousands of labelled crops that look like what the store tablets will photograph, with the same supplier-specific quirks, and you need to control exactly how hard they are. A generator gives you both.

# fields.py
import random
import numpy as np
from PIL import Image, ImageDraw, ImageFilter, ImageFont

SUPPLIERS = ["Deccan Dairy", "Shree Flour Mills", "Konkan Cocoa", "Pune Packaging"]
DEVICES = ["scanner", "phone"]
FIELD_TYPES = ["date", "amount", "invoice_no", "batch_code"]
CHARSET = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZs/-.,: "     # index 0 is reserved for the CTC blank
CHAR_TO_ID = {c: i + 1 for i, c in enumerate(CHARSET)}
H, W = 32, 128

def field_text(supplier: int, field: int, rng: random.Random) -> str:
    d, m, y = rng.randint(1, 28), rng.randint(1, 12), rng.choice([2025, 2026])
    amount = rng.randint(80, 99999) + rng.choice([0, 0.5, 0.25, 0.75])
    if field == 0:   # date: each supplier prints dates its own way
        return [f"{d:02d}/{m:02d}/{y}", f"{d:02d}-{m:02d}-{y % 100:02d}",
                f"{d:02d}.{m:02d}.{y}", f"{y}-{m:02d}-{d:02d}"][supplier]
    if field == 1:   # amount
        return [f"Rs {amount:,.2f}", f"{amount:.2f}", f"Rs.{int(amount)}", f"{amount:,.2f}"][supplier]
    if field == 2:   # invoice number
        n = rng.randint(100, 99999)
        return [f"DD-{n:05d}", f"SFM/{y}/{n % 1000:03d}", f"KC{n:05d}",
                f"PP-{n % 10000:04d}-{rng.choice('ABC')}"][supplier]
    # batch code: each supplier has its own pattern of look-alike letters (BOSIZG) and digits (805126)
    L = lambda: rng.choice("BOSIZG")
    D = lambda: rng.choice("805126")
    head = [L() + L() + L(), D() + D() + D(), L() + D() + L(), D() + L() + D()][supplier]
    return f"{head}-{rng.randint(1000, 9999)}"

def render(text: str, device: int, rng: random.Random) -> np.ndarray:
    font = ImageFont.load_default(size=rng.randint(28, 33))
    canvas = Image.new("L", (W * 2, H * 2), color=rng.randint(200, 255))   # draw at 2x, then shrink
    ImageDraw.Draw(canvas).text((rng.randint(2, 16), rng.randint(8, 22)), text,
                                fill=rng.randint(0, 70), font=font)
    if device == 1:   # phone photo: tilt and blur
        canvas = canvas.rotate(rng.uniform(-4, 4), fillcolor=230, resample=Image.BILINEAR)
        canvas = canvas.filter(ImageFilter.GaussianBlur(rng.uniform(1.5, 3.2)))
    img = np.asarray(canvas.resize((W, H), Image.BILINEAR), dtype=np.float32) / 255.0
    if device == 1:   # uneven lighting across the crop
        img = img * np.linspace(rng.uniform(0.7, 1.0), rng.uniform(0.7, 1.0), W)[None, :]
    noise = np.random.default_rng(rng.randint(0, 10**9)).normal(0, 0.02 if device == 0 else 0.08, img.shape)
    return np.clip(img + noise, 0, 1).astype(np.float32)

def make_example(rng: random.Random) -> dict:
    supplier, device, field = rng.randrange(4), rng.randrange(2), rng.randrange(4)
    text = field_text(supplier, field, rng)
    return {"image": render(text, device, rng), "text": text,
            "supplier": supplier, "device": device, "field": field}

Save a strip of examples as an image and look at them.

Working looks like: scanner crops are crisp, dark text on a light background. Phone crops are tilted, soft and grainy, and some are genuinely hard for a human to read: you might squint at a Shree Flour batch code and honestly not know whether it says 226 or ZZG. That is the point. The longest label is 12 characters (Rs 99,999.75 or SFM/2026/417).

Most likely bug: text running off the right edge of the crop, so the label contains characters the image does not. Check the widest strings at the largest font size with font.getbbox(text); the right edge must stay inside the 256-pixel canvas.

Stuck? Hint

If ImageFont.load_default(size=...) raises a TypeError, your Pillow is older than 10.1. Upgrade with uv add "pillow>=10.1", or load any TrueType font file with ImageFont.truetype(path, size).

2. Build the Dataset and a CTC-friendly collate function

Why. Labels have different lengths: KC40213 is 7 characters, Rs 12,345.75 is 12. PyTorch’s default batching wants every tensor in a batch to have the same shape, so you need your own collate function (the function a DataLoader uses to combine single examples into a batch). CTC loss wants all labels glued end to end in one long tensor, plus a list of each label’s length.

# data.py
import random
import torch
from torch.utils.data import Dataset
from fields import make_example, CHAR_TO_ID

class InvoiceFields(Dataset):
    def __init__(self, n: int, seed: int):
        rng = random.Random(seed)
        self.examples = [make_example(rng) for _ in range(n)]   # generate once, reuse every epoch

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i):
        e = self.examples[i]
        return {
            "image": torch.from_numpy(e["image"]).unsqueeze(0),            # (1, 32, 128)
            "supplier": torch.tensor(e["supplier"]),
            "device": torch.tensor(e["device"]),
            "field": torch.tensor(e["field"]),
            "target": torch.tensor([CHAR_TO_ID[c] for c in e["text"]]),   # e.g. 7 to 12 IDs
            "text": e["text"],
        }

def collate(batch):
    return {
        "image": torch.stack([b["image"] for b in batch]),                  # (batch, 1, 32, 128)
        "supplier": torch.stack([b["supplier"] for b in batch]),
        "device": torch.stack([b["device"] for b in batch]),
        "field": torch.stack([b["field"] for b in batch]),
        "targets": torch.cat([b["target"] for b in batch]),                 # all labels, end to end
        "target_lengths": torch.tensor([len(b["target"]) for b in batch]),
        "texts": [b["text"] for b in batch],                                # kept for evaluation
    }

Build three splits from different seeds: 6,000 crops for training, 2,000 for validation and 2,000 for testing. Then DataLoader(train_set, batch_size=64, shuffle=True, collate_fn=collate).

Working looks like: one batch has image of shape (64, 1, 32, 128), targets of length somewhere around 600 (64 labels of 7 to 12 characters), and target_lengths summing to exactly that length.

Most likely bug: using index 0 for a real character. CTC reserves one index for the blank, and this project uses 0. That is why CHAR_TO_ID starts at 1. If a real character maps to 0, the model can never output it.

Stuck? Hint

Generating 6,000 crops takes a few seconds. If you change the generator often, cache the generated arrays to a .npz file. Keep the test seed fixed forever, so every ablation is scored on identical crops.

3. Build the CNN image encoder

Why. A CNN turns pixels into features (strokes, curves, character parts), and with the right pooling it turns a 2-D image into a 1-D left-to-right sequence, ready for a sequence model.

# model.py
import torch
from torch import nn

def conv_block(c_in: int, c_out: int, pool) -> nn.Sequential:
    return nn.Sequential(nn.Conv2d(c_in, c_out, kernel_size=3, padding=1),
                         nn.BatchNorm2d(c_out), nn.ReLU(), nn.MaxPool2d(pool))

cnn = nn.Sequential(
    conv_block(1, 32, (2, 2)),      # (1, 32, 128)  -> (32, 16, 64)
    conv_block(32, 64, (2, 2)),     #               -> (64, 8, 32)
    conv_block(64, 128, (2, 1)),    # height only   -> (128, 4, 32)
    conv_block(128, 128, (4, 1)),   # height only   -> (128, 1, 32)
)

x = torch.rand(8, 1, 32, 128)
f = cnn(x)                           # (8, 128, 1, 32)
seq = f.squeeze(2).permute(0, 2, 1)  # (8, 32, 128): 32 steps, 128 features each
print(f.shape, seq.shape)

squeeze(2) removes the height dimension, which is now 1. permute(0, 2, 1) swaps the last two dimensions so time (the 32 columns) comes before features, matching batch_first=True for the LSTM.

Why 32 steps? CTC needs at least one step per character, plus one for each pair of identical neighbours (milestone 5 explains why). The longest label is 12 characters, so 32 leaves room, and each step’s receptive field is wide enough to see a whole character.

Working looks like: torch.Size([8, 128, 1, 32]) and torch.Size([8, 32, 128]).

Most likely bug: pooling the width too aggressively, for example (2, 2) in all four blocks. You end up with 8 steps for 12-character labels, and CTC returns infinite losses (milestone 6’s zero_infinity=True then silently hides it as zero). Always check that steps ≥ 2 × longest label.

Stuck? Hint

Write the expected shape after each block in a comment, as above, then verify with a random input before you build anything else on top. Shape mistakes are much cheaper to find here than inside the loss.

4. Add the metadata encoder and fuse the streams

Why. The supplier tells the model which formats and which characters to expect. To be useful at every character, the metadata must be available at every step of the sequence, not bolted on at the end.

# model.py (continued)
from fields import CHARSET, SUPPLIERS, DEVICES, FIELD_TYPES

class InvoiceReader(nn.Module):
    def __init__(self, use_metadata: bool = True):
        super().__init__()
        self.use_metadata = use_metadata
        self.cnn = cnn_encoder()                                   # the 4-block CNN from milestone 3
        self.supplier_emb = nn.Embedding(len(SUPPLIERS), 8)
        self.device_emb = nn.Embedding(len(DEVICES), 4)
        self.meta_mlp = nn.Sequential(nn.Linear(8 + 4, 32), nn.ReLU())
        self.rnn = nn.LSTM(128 + 32, 128, batch_first=True, bidirectional=True)
        self.text_head = nn.Linear(256, len(CHARSET) + 1)          # +1 for the blank
        self.field_head = nn.Linear(256, len(FIELD_TYPES))

    def forward(self, image, supplier, device):
        f = self.cnn(image).squeeze(2).permute(0, 2, 1)            # (batch, 32, 128)
        meta = self.meta_mlp(torch.cat([self.supplier_emb(supplier), self.device_emb(device)], dim=1))
        if not self.use_metadata:
            meta = torch.zeros_like(meta)                          # ablation switch
        meta = meta.unsqueeze(1).expand(-1, f.shape[1], -1)        # (batch, 32, 32): copy to every step
        seq, _ = self.rnn(torch.cat([f, meta], dim=2))             # fuse -> (batch, 32, 256)
        return {"text": self.text_head(seq),                       # (batch, 32, 45) per-step scores
                "field": self.field_head(seq.mean(dim=1))}         # (batch, 4)

(Wrap the CNN from milestone 3 in a small function cnn_encoder() that returns a fresh copy, so every model gets its own weights.)

The fusion is two lines: unsqueeze(1) and expand repeat the metadata along the time dimension without copying memory, and torch.cat(..., dim=2) glues it onto each step’s image features. The use_metadata switch is for milestone 8; building the ablation into the model guarantees the “without metadata” version is otherwise identical.

Working looks like: for a batch of 8, out["text"] is (8, 32, 45) (44 characters plus blank) and out["field"] is (8, 4). The model has 550,712 parameters: 240,960 in the CNN, 296,960 in the BiLSTM, about 12,000 in the two heads, and fewer than 500 in the metadata encoder. The metadata path is tiny; its value, if any, comes from what it says, not how big it is.

Most likely bug: concatenating metadata only to the final pooled vector for the field head and not to the sequence. Then the text head never sees the supplier at all.

Stuck? Hint

If torch.cat complains about sizes, print f.shape and meta.shape right before it. They must agree on every dimension except the last.

5. Add the CTC text head and the auxiliary field-type head

Why. This is the conceptual heart of the project. You know the text in each crop, KC40213, but not where each character is. Which of the 32 steps is the K? Labelling that by hand for thousands of crops would be absurd. CTC (connectionist temporal classification) lets the model learn reading without any position labels.

Priya’s analogy: transcribing a song when you know the lyrics but not the timing. You do not need a timestamp for each word. You only need to agree, afterwards, on a rule for turning “what was heard at each moment” into the lyrics.

CTC’s rule has two steps. At each of the 32 steps, the model outputs one symbol: a character or a blank (written _ here, “nothing new at this step”). Then decode:

1. Collapse repeats: consecutive identical symbols merge into one.
2. Remove blanks.

A worked example with 12 steps for the label KC40:

Step	1	2	3	4	5	6	7	8	9	10	11	12
Output	_	K	K	_	C	_	_	4	_	0	0	_

Collapse repeats: _ K _ C _ 4 _ 0 _. Remove blanks: KC40. Many different paths decode to the same text: KKK_CC_4_00_, _KC__4___0__ and hundreds more. CTC loss adds up the probability of every path that decodes to the right label, using an efficient dynamic-programming algorithm, and trains the model to make that total probability high. The model is free to put each character wherever the image says it is.

Why is the blank essential? Take a Konkan invoice number KC00093. Without blanks, the two zeros 0 0 would collapse into one 0. With a blank between them, 0 _ 0 survives as 00. That is also why you need extra steps for repeated characters.

In PyTorch:

ctc_loss = nn.CTCLoss(blank=0, zero_infinity=True)
ce_loss = nn.CrossEntropyLoss()

out = model(batch["image"], batch["supplier"], batch["device"])
log_probs = out["text"].log_softmax(2).permute(1, 0, 2)          # CTC wants (steps, batch, symbols)
steps, n = log_probs.shape[0], log_probs.shape[1]
input_lengths = torch.full((n,), steps, dtype=torch.long)       # every crop uses all 32 steps

loss_text = ctc_loss(log_probs, batch["targets"], input_lengths, batch["target_lengths"])
loss_field = ce_loss(out["field"], batch["field"])

Four details catch everyone. CTC wants log-probabilities, so apply log_softmax. It wants time first, (steps, batch, symbols), hence the permute. It wants the concatenated targets and their lengths from your collate function. And zero_infinity=True replaces impossible alignments (a label longer than the steps allow) with zero loss instead of crashing; useful insurance, but it can hide a shape bug, which is why milestone 3 checked the step count.

Working looks like: on an untrained model, the CTC loss is large, typically 10 to 30 per example (PyTorch averages it per target character by default, so you will see a few units per batch at the very start). The field cross-entropy starts near $\ln 4\approx 1.39$.

Most likely bug: passing raw scores instead of log-probabilities, or forgetting the permute. Both produce a loss that decreases a little and then stalls, with decoded outputs that are empty or a single repeated character.

Stuck? Hint

Test the loss on a tiny hand-made case: 1 example, 5 steps, target [3]. Make log-probabilities that put almost all the mass on symbol 3 at step 2 and blank elsewhere. The loss should be close to zero. Then move the mass to the wrong symbol; it should jump.

6. Train with a weighted multi-head loss

Why. Two heads, two losses, one .backward(). As in the multi-IO lesson, you must decide how much each counts. Here the auxiliary loss is a helper, so it gets a weight $\lambda$ (lambda) that you will vary.

$$L=L_{\text{CTC}}+\lambda L_{\text{field}}$$

Both losses are already on comparable scales (around 0.1 to 3 during training), so no rescaling is needed. That is worth checking rather than assuming: log both separately from the first epoch.

# train.py
import sys
import torch
from torch import nn
from torch.utils.data import DataLoader
from data import InvoiceFields, collate
from model import InvoiceReader
from evaluate import evaluate                      # milestone 7

lam = float(sys.argv[1]) if len(sys.argv) > 1 else 0.3
use_meta = (sys.argv[2] != "nometa") if len(sys.argv) > 2 else True
device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"

train_dl = DataLoader(InvoiceFields(6000, seed=1), batch_size=64, shuffle=True, collate_fn=collate)
val_dl = DataLoader(InvoiceFields(2000, seed=2), batch_size=256, collate_fn=collate)

torch.manual_seed(0)
model = InvoiceReader(use_metadata=use_meta).to(device)
ctc_loss, ce_loss = nn.CTCLoss(blank=0, zero_infinity=True), nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-4)
best_cer, best_state = float("inf"), None

for epoch in range(1, 16):
    model.train()
    sums = {"ctc": 0.0, "field": 0.0}
    for b in train_dl:
        out = model(b["image"].to(device), b["supplier"].to(device), b["device"].to(device))
        log_probs = out["text"].log_softmax(2).permute(1, 0, 2)
        steps, n = log_probs.shape[:2]
        l_ctc = ctc_loss(log_probs, b["targets"].to(device),
                         torch.full((n,), steps, dtype=torch.long), b["target_lengths"])
        l_field = ce_loss(out["field"], b["field"].to(device))
        loss = l_ctc + lam * l_field
        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_norm_(model.parameters(), 5.0)     # LSTMs like a seatbelt
        optimizer.step()
        sums["ctc"] += l_ctc.item()
        sums["field"] += l_field.item()
    cer, exact, field_acc = evaluate(model, val_dl, device)[:3]
    print(f"epoch {epoch:2d}  ctc {sums['ctc'] / len(train_dl):.3f}  field {sums['field'] / len(train_dl):.3f}  "
          f"val CER {cer:.2%}  exact {exact:.1%}  field acc {field_acc:.1%}")
    if cer < best_cer:                                          # keep the best epoch by validation CER
        best_cer = cer
        best_state = {k: v.clone() for k, v in model.state_dict().items()}

model.load_state_dict(best_state)
torch.save(model.state_dict(), f"reader_lam{lam}_{'meta' if use_meta else 'nometa'}.pt")

The checkpoint is chosen by validation character error rate (milestone 7), the metric that matters to Meera, not by total loss.

Working looks like (λ = 0.3, with metadata):

epoch  1  ctc 3.410  field 1.394  val CER 100.00%  exact 0.0%   field acc 21.4%
epoch  3  ctc 2.111  field 1.111  val CER 49.10%   exact 0.8%   field acc 69.2%
epoch  4  ctc 0.872  field 0.608  val CER 14.89%   exact 40.5%  field acc 90.3%
epoch  9  ctc 0.100  field 0.016  val CER 4.26%    exact 76.2%  field acc 99.2%
epoch 13  ctc 0.029  field 0.003  val CER 3.79%    exact 78.6%  field acc 99.3%
epoch 15  ctc 0.022  field 0.016  val CER 5.67%    exact 70.0%  field acc 96.3%

For the first two epochs the model outputs nothing but blanks, so the CER is 100%: predicting “nothing here” everywhere is the cheapest early guess under CTC. Then, around epoch 3 or 4, it “clicks”, much like the LSTM’s sudden jump in the proofing-cabinet lesson. The field-type head reaches near-perfect accuracy quickly, because a date looks nothing like a batch code. And look at epoch 15: validation CER rose to 5.7% after a noisy final epoch. That is exactly why the loop keeps the best checkpoint (epoch 13 here) rather than the last one.

Most likely bug: the field loss stuck at 1.386 ($\ln 4$) while CTC falls. With λ = 0 that is expected (the head gets no training signal). With λ > 0 it means the field head is not connected to the loss, usually because loss = l_ctc was left over from debugging.

Stuck? Hint

On Apple Silicon, if you get an error that an operation is not implemented for the MPS device, compute the CTC loss on the CPU: ctc_loss(log_probs.cpu(), b["targets"], ...). Gradients still flow back to the model on the GPU.

7. Decode and evaluate with character error rate

Why. The model outputs 32 rows of scores; Meera needs strings. And “accuracy” needs care for text: is KC4O213 wrong, or one character wrong?

Greedy decoding takes the highest-scoring symbol at each step, then applies the CTC rule (collapse repeats, drop blanks):

# evaluate.py
import numpy as np
import torch
from torchmetrics.text import CharErrorRate
from fields import CHARSET

def greedy_decode(logits: torch.Tensor) -> list[str]:
    """logits: (batch, steps, symbols) -> list of strings."""
    texts = []
    for path in logits.argmax(dim=2).tolist():
        chars, prev = [], 0
        for k in path:
            if k != prev and k != 0:          # new symbol, and not a blank
                chars.append(CHARSET[k - 1])
            prev = k
        texts.append("".join(chars))
    return texts

@torch.no_grad()
def evaluate(model, loader, device):
    model.eval()
    preds, truths, field_ok, n = [], [], 0, 0
    for b in loader:
        out = model(b["image"].to(device), b["supplier"].to(device), b["device"].to(device))
        preds += greedy_decode(out["text"].cpu())
        truths += b["texts"]
        field_ok += (out["field"].argmax(1).cpu() == b["field"]).sum().item()
        n += len(b["texts"])
    cer = CharErrorRate()(preds, truths).item()
    exact = float(np.mean([p == t for p, t in zip(preds, truths)]))
    return cer, exact, field_ok / n, preds, truths

Character error rate (CER) is the number of single-character edits (insertions, deletions, substitutions) needed to turn the prediction into the truth, divided by the length of the truth. KC4O213 for KC40213 is one substitution in 7 characters: CER 14%. Report it alongside exact match, the share of crops read perfectly, because for a batch code, one wrong character makes the whole code useless.

Now trace one hard crop from the test set. It is a phone photo of a Pune Packaging batch code, 2S2-2153: digit, letter, digit, then four digits. Here is the most likely symbol at each of the 32 steps, from the trained model with metadata:

2__S_2_-2_11_5_3________________

Collapse the repeats (the 11 becomes one 1) and remove the blanks, and you get 2S2-2153. Exactly right. The model reads the eight characters in the first sixteen steps and spends the rest of the crop, which is empty background, on blanks. Notice how the model “fires” each character at one or two steps and fills the gaps with blanks, without anyone ever telling it where the characters were.

The same crop, read by the identical model trained without metadata: 25S2-2153, an extra character inserted where the blurry 2 and S run together. And by the metadata model, but told (wrongly) that the invoice came from Deccan Dairy, whose batch codes are three letters: ZSZ-2153. Every 2 became a Z. Same pixels, different supplier, different reading. The model has learned that the supplier changes what a blurry shape most likely is, exactly the behaviour Priya predicted.

Working looks like: on the 2,000 test crops, overall CER of a few percent, almost all of it on phone photos, and most of that on batch codes.

Most likely bug: decoding with CHARSET[k] instead of CHARSET[k - 1]. Every character comes out shifted by one (L instead of K), and CER is close to 100% while the loss looks perfectly healthy.

Stuck? Hint

Print a dozen (truth, prediction) pairs before computing any metric. Patterns jump out: all wrong in the same way means a decoding bug; wrong only on blurry phone batch codes means a modelling limit.

8. Run ablations on loss weight and metadata

Why. You added two design choices on faith: metadata fusion and an auxiliary head. An ablation removes or varies one thing at a time, keeping everything else identical (same data, same seed, same epochs, same test set), to measure what each choice is actually worth. Without this step, the architecture diagram is a guess.

Run five configurations:

uv run python train.py 0 meta       # no auxiliary loss
uv run python train.py 0.3 meta     # small auxiliary weight
uv run python train.py 1.0 meta
uv run python train.py 3.0 meta
uv run python train.py 0.3 nometa   # metadata switched off

Then score each saved model on the same 2,000 test crops, broken down by device and by field type. Arjun’s results:

Configuration	Test CER	Exact match	Scanner exact	Phone exact	Phone batch-code exact
Auxiliary weight 0, with metadata	4.19%	77.6%	98.1%	58.6%	42.8%
Auxiliary weight 0.3, with metadata	4.21%	77.0%	97.8%	57.7%	42.8%
Auxiliary weight 1.0, with metadata	3.93%	78.0%	97.7%	59.9%	40.6%
Auxiliary weight 3.0, with metadata	5.07%	72.5%	97.0%	50.0%	33.2%
Auxiliary weight 0.3, no metadata	4.87%	74.7%	97.7%	53.4%	33.6%

Read it column by column, the way Priya made Arjun do it.

Scanner crops are essentially solved (about 98% exact in every configuration). Every interesting difference lives in the phone photos, which is where the effort should go.

Metadata earns its place. Switching it off (last row against the 0.3 row) raises overall CER from 4.21% to 4.87%, and on the hardest slice, phone photos of batch codes, exact reads fall from 42.8% to 33.6%: nearly a quarter of the correct reads disappear. That is the SBO versus 580 problem, measured. Keep it.

The auxiliary head does not earn its place here. Weights of 0, 0.3 and 1.0 land within about 0.3 percentage points of CER of each other, which is within the noise of a single run. A weight of 3.0 actively hurts: in its log, the field loss dominated the early epochs and CTC only “clicked” around epoch 8 instead of epoch 4, so with the same 15-epoch budget it finished worse. The field type is easy to learn and already known from the template, so it teaches the shared layers little that CTC does not. Conclusion: remove it, or keep it at a small weight only if you want it as a sanity check that the template cropped the right field.

That second conclusion is not a failure of the project. It is the project working. You built a component on a plausible theory, tested it properly, and found it unnecessary. Auxiliary losses do help in many systems, usually when the main task’s signal is weak or labels are scarce; here, CTC had plenty of signal.

Phone batch codes are still poor, at around 43% exact. No loss weighting fixes blur that destroys the evidence. The practical fixes are outside the model: better capture (an app overlay that asks for a steady, well-lit photo), more realistic training data, and format checks that send doubtful reads to a person (see the stretch goals).

Working looks like: a table like this one, with your own numbers, and a written conclusion for each design choice: keep, remove or “no measurable effect”.

Most likely bug: comparing runs with different test sets or different seeds and reading noise as signal. Fix the test seed, fix the training seed, and if two numbers differ by less than about half a percentage point of CER, rerun with another seed before believing the difference.

Stuck? Hint

Save each run’s test predictions to a CSV (truth, prediction, supplier, device, field). Then every breakdown is a one-line filter, and you can show Meera the actual crops each model got wrong.

Done means done

• fields.py `fields.py` generates crops for four suppliers and two devices, and you have looked at at least 20 of them.
• The collate function returns concatenated targets and lengths, and the blank is index 0.
• The CNN turns (batch, 1, 32, 128) into (batch, 32, 128) The CNN turns `(batch, 1, 32, 128)` into `(batch, 32, 128)`, checked with a random input.
• Metadata is fused at every time step, with a use_metadata Metadata is fused at every time step, with a `use_metadata` switch.
• Training logs CTC loss and field loss separately, and keeps the best checkpoint by validation CER.
• Greedy decoding works and you can explain one decoded path step by step.
• Test CER and exact match are reported overall, by device and by field type.
• Five ablation runs are in one table, each with a one-sentence conclusion.

Stretch goals

• Format validation. Check each decoded field against its supplier’s format with a regular expression (SFM/\d{4}/\d{3}, a date that parses) and route failures to a human with the crop shown.
• Beam search decoding. Keep the top few candidate paths instead of the single best symbol per step, and combine them with the format rules.
• Train on real crops. Label 50 real invoices’ fields, fine-tune on them, and measure CER on held-out real crops before and after.
• The recall test. Read a month of synthetic invoices into SQLite and time the query “which stores received batch SBO-4412?”. Compare with two days.

Common pitfalls

• CTC plumbing. Too few time steps, the wrong tensor layout, or raw scores instead of log-probabilities all produce a loss that falls deceptively. zero_infinity=True is insurance, not a fix.
• Choosing the checkpoint by total loss. Choose by the metric the business cares about, here validation CER.
• Trusting synthetic results on real invoices. One built-in font is not real fonts, stamps, folds and handwriting. Synthetic data builds and debugs the system; real labelled crops are needed before anyone relies on it.
• Skipping the ablation. Every component should earn its place with a number.

Show it off

Make the README lead with the trace: one blurry phone crop, its 32-step path, the decoded text, and the different readings with and without the supplier. That single figure explains CTC, fusion and why metadata matters better than any paragraph.

Then the ablation table, with honest conclusions, including any component that did not help. Engineers reviewing portfolios are far more impressed by “I measured it and removed it” than by a diagram with every box they have heard of.

For LinkedIn: “I built a multi-input OCR model that reads invoice fields by fusing a CNN over the image with the supplier’s identity, trained with CTC loss. Ablations showed the supplier input lifted exact reads of blurry phone batch codes from 34% to 43%, while an auxiliary head I had expected to help made no measurable difference, so I removed it.” Add the trace figure.

A week later, Deccan Dairy sent a routine notice about a different batch. Meera typed the code into the new search box. Three stores, two delivery dates, the pastry lines that used it, in under a second. She did not say anything for a moment.

“Two days, last time,” she said finally.

“Scanned invoices it reads right 98% of the time,” Arjun said. “Phone photos are worse, especially batch codes, so anything that fails the format check goes to accounts with the photo next to it. And the tablet app now tells staff to hold the invoice flat under the counter light before it lets them take the picture.”

“So people still check some of them.”

“About one phone photo in four. Instead of typing all four hundred invoices a month.”

“That I can live with.” Priya was already sketching on the whiteboard: a sequence of words, each one drawing arrows to every other. “Next module, we stop passing notes down a line of LSTM cells. Every word gets to look at every other word directly. That’s attention, and it’s how the models you used all of Part 1 actually work.”

Go deeper

• PyTorch tutorials and the API reference for nn.CTCLoss, which documents every input shape.
• torchmetrics documentation for CharErrorRate and other text metrics.
• Attention Is All You Need, for where sequence models went next, in Module 30.