Scan Evaluation Protocol

This page defines how we evaluate scanner speed and accuracy across the mobile app, backend endpoints, and future sealed/slab flows. The goal is not "the model felt good once"; the goal is a repeatable protocol that tells us which path is fast, which path is correct, and which UX fallback made the user successful.

Scope

Current mobile fan-out paths:

Mode	Input	Endpoint	Purpose
lab	Raw camera JPEG	/scan-lab	Main pgvector/DINOv2 path over the backend catalog
compat	Raw camera JPEG	/scan-compat	In-stack tcg-vision FAISS comparison path
tormknd	Raw camera JPEG	Tormknd /scan	Upstream reference path
combined	Raw camera JPEG	/scan	Historical multi-signal path: pHash + OCR + embedding
frame	Client-warped crop	/scan-frame	Experimental Path #5: phone-side OpenCV crop before backend embedding

Future modes must fit this same protocol before they become product candidates: slabs, sealed products, graded labels, and any OCR-first hybrid path.

Definitions

• Top-1 accuracy: the first displayed match is exactly the expected item.
• Top-3 recoverability: the expected item appears in the endpoint's alternatives or debug topK when available.
• User-success accuracy: the user can finish the flow correctly through top match, alternatives, or correction UI. This is the product target; it can approach 100% even when raw top-1 cannot.
• Client latency: wall-clock time observed by the phone around upload/prep/network.
• Server latency: backend-reported ms inside the response.
• First useful result: first endpoint returning ok: true with a non-null match.
• Blocking path: an endpoint that delays the scanner state after another endpoint already produced a useful result.

Metrics Per Capture

Every captured photo should record:

Metric	Source	Why it matters
Capture timestamp	Scan Debug entry	Groups the five endpoint rows
Active TCG filter	Scan Debug entry	Explains forced ?tcg= behavior
Endpoint mode	Scan Debug entry	Makes A/B comparison possible
Client elapsed time	clientTotalMs or clientMs	What the user actually waited for
Server elapsed time	response ms	Separates backend cost from phone/network cost
Frame prep timings	clientPrepMs, clientDetectMs, clientWarpMs	Isolates Path #5 phone-side cost
Result state	match, no-match, failure reason	Separates accuracy from availability
Confidence/similarity	response fields	Helps tune thresholds and ranking
Archived case path	backend Scan Debug case archive	Lets reviewers inspect the exact input, crop, top-K, and backend JSON later
User verdict	future correction UI	Required for real accuracy learning

Dataset Tiers

Start small, but keep the tiers explicit.

Tier	Size	Contents	Use
Smoke	5-10 images	Known easy cards, one real phone capture each	Validate a build before asking for more phone tests
Lab	50 images	Easy cards, visually similar cards, glare, tilt, poor background	Compare endpoint quality and latency
Regression	200+ images	Every historical wrong match and timeout	Prevent known failures from coming back
Product	500+ images	Cards, slabs, sealed, multiple languages, multiple devices	Release gate for scanner UX

Each dataset row needs:

• image_id
• expected item id
• item type: card, slab, sealed
• TCG
• language
• set / product family
• variant notes: holo, reverse, promo, graded company, grade, sealed product type
• capture context: device, lighting, background, distance, tilt
• expected acceptable alternatives, if any

Phone Test Runbook

1. Pull latest poc-tcg-collector and backend repos.
2. Confirm exposed endpoints respond with a known image before opening the app.
3. Build/install the Android app on the development machine.
4. Clear Scan Debug history.
5. Scan each physical item three times:
   • normal centered capture
   • slight tilt
   • lower-light or glare capture
6. For every bad result, keep the Scan Debug group expanded and confirm the case was archived.
7. Save a screenshot only when the UI layout itself is the bug; the backend case archive is the source of truth for recognition review.
8. Record whether the user could still finish correctly through alternatives or correction.

Scan Debug Case Archive

The app can upload Scan Debug cases to the backend lab archive when the local upload config is enabled. The archive stores the input image, the backend result JSON, top-K candidates, latency fields, and the private crop artifact when debug preprocessing returned one.

Treat this archive like a machine-learning intake queue, not a training set. A case becomes useful only after a human labels the expected item, marks whether the displayed match was correct, and classifies the failure mode. Labelled cases can then be copied into the appropriate benchmark folder, such as sealed-eval, when rights are clear.

After a phone test session, run the backend review command:

cd tcg-collector-backend/src/api
npm run scan-debug:review -- \
  --markdown /tmp/scan-debug-review.md \
  --json /tmp/scan-debug-review.json

The command checks every archived case folder, verifies that input.jpg and metadata.json exist, detects optional crop.jpg, counts ambiguous responses and missing labels, and writes a Markdown report for human review.

The internal Cardex backend app also has a protected scan debug section. It reads the same archive through GET /scan-debug-cases, then fetches input.jpg and crop.jpg through bearer-auth artifact URLs. Use this screen for fast visual review after an associate test batch: compare the uploaded input, the crop, the scanner match, the reason, and the top-K IDs before adding the human verdict.

Backend Smoke Command

Use this only as a backend availability smoke test. It does not replace phone testing because it does not exercise camera capture, Android networking, or client-side Frame prep.

img=/home/kael/skn-tcg/scan-debug.jpeg
curl -sS -m 12 -w "lab http=%{http_code} total=%{time_total}s\n" \
  -F "photo=@$img;type=image/jpeg" http://127.0.0.1:8080/scan-lab
curl -sS -m 12 -w "frame http=%{http_code} total=%{time_total}s\n" \
  -F "photo=@$img;type=image/jpeg" http://127.0.0.1:8080/scan-frame

For Tormknd, the multipart field is file, not photo.

Decision Rules

• Keep all five endpoints while we are comparing scanner behavior.
• Do not promote a path because it wins one screenshot; promote it when it wins the dataset tier.
• If an endpoint is slow but uniquely correct, keep it as a secondary/correction candidate.
• If an endpoint is fast but often wrong, keep it visible in lab mode only until ranking is fixed.
• If frame times out often and the four raw-photo endpoints already identify correctly, make Frame opt-in or keep it debug-only.
• Measure user-success separately from raw top-1. The product can be excellent if the UI makes correction fast and learns from it.

Current Baseline Note

On 2026-05-13, a VPS smoke against scan-debug.jpeg returned roughly:

Endpoint	curl total
/scan-frame	0.62 s
/scan-lab	0.83 s
/scan combined	0.84 s
Tormknd /scan	1.26 s
/scan-compat	1.42 s

This suggests that the visible Path #5 slowdown should first be investigated in the Android client-side OpenCV prep path, not in /scan-frame itself. Phone captures remain the source of truth.