Documentation

Appearance

Path #5 "Frame" — alternatives considered ​

Design doc written on 2026-05-11, before implementation. Captures every path we evaluated (not just the chosen one) so we keep a trail of the trade-offs. If we revisit the decision in 6 months, this document explains why each option was kept or dropped.

Context ​

The POC scanner currently exposes 4 comparable paths, all toggleable from the mobile app:

PathEndpointTypePrimary signal
Lab/scan-labmono-signalDINOv2 + pgvector (HNSW)
Compat/scan-compatmono-signalDINOv2 + FAISS (fork)
Tormknd/scan-tormkndmono-signalDINOv2 + FAISS (upstream gold standard)
Combined/scanmulti-signalpHash + Tesseract OCR + DINOv2

All four receive the raw photo taken from the app, and the backend (tcg-vision) handles quad detection, perspective correction and embedding. As a consequence, the user's framing quality dictates the entire pipeline, and the UX boils down to "point and pray".

Path #5's goal is to introduce real-time frame detection on the client — the familiar green box that snaps to the card live, as in Ludex, Collectr, Dragon Shield Card Manager, Adobe Scan or Apple Notes. This document recaps the options evaluated before locking in V1.

Three possible axes to differentiate Path #5 ​

Path #5 needs to stand apart from the existing 4 paths enough to make the A/B meaningful. Three axes surfaced during design:

Axis 1 — Real-time client-side pre-crop ​

The client (RN + native plugin) detects the quad on every preview frame, the user sees the box snap to the card live, and the app only uploads an already-rectified crop (perspective corrected) to the backend. The matcher remains one of the existing ones.

  • Backend endpoint: reuse /scan-lab or new /scan-frame that bypasses the server-side warp.
  • Measurable difference: input quality. If the client crop is better than the server crop, the matcher returns a higher score.
  • Cost: Android native plugin (Kotlin + OpenCV) + Skia warp on the client.

Axis 2 — Different matcher ​

Keep the current input (raw photo or crop) but plug in a matcher of a different nature:

  • ORB / SIFT + RANSAC — geometric feature matching. Complementary to DINOv2 (which is semantic). Robust to varied backgrounds but requires a separate index (~5 GB for 150 k cards).
  • OCR-first (Tesseract) — on a clean crop, the card name and number are readable at 95 %+. Direct DB lookup instead of similarity search.

Axis 3 — Fully on-device ​

Detection + matching local, no backend. The app downloads:

  • pHashes for every card (BIGINT, ~1.2 MB for 150 k cards),
  • optionally the embeddings (DINOv2 384-d × 150 k = ~230 MB, large for mobile),
  • or an embedded model (TFLite, quantized).

Upside: zero network latency, works offline. Downside: index frozen at download time, expensive to refresh.

V1 choice — Axis 1 only. Axis 1 isolates a single variable (crop quality) cleanly and enables a clean A/B vs /scan-lab. Axis 2 will come in V2 if Axis 1 confirms the hypothesis. Axis 3 stays a separate project (existing TODO around /cards/dump + phash column).

Native Android detection — three possible implementations ​

Once Axis 1 is chosen, we still need to pick which native library detects the quad on the client.

Option A — OpenCV Android SDK ✅ chosen ​

Canny → findContours → approxPolyDP (epsilon = 0.02 × perimeter)
       → filter 4 vertices + convexity + ratio [0.65, 0.80]
       → stability tracking (5 frames, drift < 8 px)
  • Precision: full control over Canny thresholds, polygon approximation epsilon, aspect ratio filter. Same parameters as tcg-vision/src/vision_service.py on the backend, which guarantees parity.
  • APK weight: ~30 MB (native libs for arm64-v8a + armeabi-v7a). With arm64-v8a only, ~15 MB.
  • Maintenance: OpenCV is mature, stable, and the team already has expertise (the backend has used it from day one).

Option B — Google ML Kit Document Scanner ​

Plug-and-play Google API on top of ML Kit Vision.

  • Pros: only ~5 MB, one-line API (DocumentScannerOptions.Builder().build()), maintained by Google, works on Android 7+.
  • Blocker: tuned for A4 documents. The internal thresholds (minimum size, expected ratio) are calibrated for paper sheets, not 63×88 mm TCG cards. In real conditions (wooden table, varied background), it tends to detect the table rectangle rather than the card.
  • Verdict: worth a quick spike but unreliable for this use case.

Option C — TensorFlow Lite + custom model ​

Train an object detection model (e.g. SSD MobileNet or YOLOv8n) on a dataset of annotated cards (bounding box + 4 corners).

  • Pros: robust on complex backgrounds (cards in a binder, under a sleeve, etc.) where Canny fails. Latency ~15 ms on mobile GPU.
  • Cons: need to build the dataset (≥ 1 000 annotated images across the 6 supported TCGs, ~3 days of work), train, quantize, validate. So ~1 extra week on top of V1.
  • Verdict: a V2 candidate if OpenCV hits a ceiling on hard conditions.

V1 matcher choice — three options ​

Path #5 ships a pre-warped crop to /scan-frame. Which matcher runs behind it?

Option A — Reuse card_embeddings_lab (DINOv2) ✅ chosen ​

/scan-frame shares the card_embeddings_lab index with /scan-lab. The only difference between the two paths is where the crop comes from: server warp for Lab, client warp for Frame.

  • Clean A/B: if /scan-frame beats /scan-lab on a real scan set, we have proof that the client crop improves embedding quality — nothing else changed.
  • Zero new index to build, index, maintain, re-embed.
  • Risk: if the client crop is not meaningfully better, the A/B is a wash and Path #5's added value is purely UX (live frame). Still a real gain but harder to measure.

Option B — ORB features + RANSAC (new matcher) ​

Build a separate ORB index (binary keypoint descriptors) and match via RANSAC for homography transforms.

  • Pros: pure geometric signal, complementary to DINOv2 (semantic). Excellent for distinguishing visually similar cards with different text (reprints, alt-arts).
  • Cons: new index to build (~500 keypoints × 32 bytes × 150 k cards = ~2.4 GB), extraction pipeline on the ingest side, +2 days of work, doubles the A/B (we isolate the crop and the matcher → can't tell their respective contributions apart).
  • Verdict: a V2 candidate as a complementary matcher in /scan (multi-signal), not as a Path #5 replacement.

Option C — Tesseract OCR on a clean crop ​

On a pre-warped clean crop, Tesseract reaches > 95 % accuracy on the card name and number. Direct lookup in cards.name or cards.card_number.

  • Pros: very fast (~100 ms), no embedding, deterministic.
  • Cons: this is no longer "similarity matching" but text resolution. Comparable to the other 4 paths only on overall hit rate, not on score semantics. Breaks the "5 matchers of the same nature" debug dashboard narrative.
  • Verdict: Tesseract stays inside /scan (Combined) as a complementary signal; no dedicated Path.

Platform scope — V1 ​

Option A — Android-only in V1 ✅ chosen ​

Main development and user testing happens on Pixel (Android). Swift plugin will land in V2 if the feature proves its value.

  • Gain: -2 days on V1, ships faster, lets us learn before investing in iOS.
  • Acceptable because the app is a POC, not a public release.

Option B — Android + iOS simultaneously ​

Develop the Kotlin plugin (OpenCV) and the Swift plugin (Vision framework, VNDetectRectanglesRequest) in parallel, with the RN overlay shared.

  • Cost: +2 days, doubles the bug surface, forces maintenance of 2 native implementations from day one.

Option C — iOS first ​

The Apple Vision framework (VNDetectRectanglesRequest) is a one-line API, simpler to integrate than OpenCV. But since testing happens mainly on Android, low practical value.

Final decision ​

DecisionChosenDiscarded
Differentiation axisAxis 1 — client-side pre-crop in real timeAxis 2 (different matcher), Axis 3 (on-device)
Native Android libOpenCV Android SDKML Kit Document Scanner, custom TFLite
V1 matchercard_embeddings_lab (DINOv2 reused)ORB+RANSAC, Tesseract OCR
V1 platformAndroid onlyAndroid+iOS simultaneous, iOS first

Implementation plan: 4 phases, ~13 PRs, ~5 working days. Details in scan-frame-plan.md (to be created at the start of implementation).

Potential V2 follow-ups (not committed) ​

  • ORB + RANSAC as a complementary matcher inside /scan (multi-signal) — break DINOv2's exclusive embedding role.
  • iOS Swift plugin once the V1 Android version is validated in the field.
  • Custom TFLite model if OpenCV hits a ceiling on complex backgrounds (cards in a binder, under a sleeve, poor lighting).
  • Fully on-device path (Axis 3) — tied to the existing /cards/dump + phash column TODO.