Card scan architecture

End-to-end view of the card-recognition pipeline on the backend, who hosts what, and how to add a new TCG without depending on anyone.

Snapshot dated 2026-05-07.

TL;DR

The React Native app scans via POST /scan-compat against tcg-vision, a Docker sidecar hosted on our VPS.
tcg-vision is built from the open-source repo SKN/tgc-app (Tormknd) but runs 100% on our infrastructure. No outbound runtime calls to Tormknd.
The only artefact we didn't generate ourselves is faiss_index.bin (~30 370 pre-computed DINOv2 embeddings). It's baked into our image at build time.
We also have a second parallel pipeline that's entirely ours (pgvector + embedCatalog.ts + /scan on the Fastify API) — currently dormant, the card_embeddings table is empty.
Magic the Gathering (Scryfall, ~85 k printings, ingested 2026-05-07) runs on the pgvector pipeline because tcg-vision has no FAISS index for MTG. See docs/INGEST.md §Magic the Gathering.
To add yet another TCG, two options: ask Tormknd to rebuild his index, OR activate our pgvector and embed ourselves (the MTG ingest is the reproducible precedent).

Diagram

┌──── Phone (RN app) ─────────────────────────────────────┐
│  ScannerScreen → backendScan.ts                         │
│  POST {photo} → http://<vps>:8000/scan-compat           │
└────────────────────────┬────────────────────────────────┘
                         │
                         ▼
┌──── Our VPS — docker compose ───────────────────────────┐
│                                                         │
│  ┌─ tcg-vision (port 8000) ──────────────────┐          │
│  │  Code  : SKN/tgc-app pinned commit        │          │
│  │  ML    : DINOv2 small (HuggingFace)       │ ← live   │
│  │  Index : faiss_index.bin (~30 370 cards)  │          │
│  │  Routes: /scan-compat, /scan              │          │
│  └─────────────┬─────────────────────────────┘          │
│                │ enriches via /cards/:id (PR #97)       │
│                ▼                                        │
│  ┌─ api Fastify (port 8080) ─────────────────┐          │
│  │  /cards/:id, /sets, /cards/dump…          │          │
│  │  /scan (pHash + dormant pgvector branch)  │ ← ready  │
│  └─────────────┬─────────────────────────────┘          │
│                ▼                                        │
│  ┌─ postgres (pgvector/pg16) ────────────────┐          │
│  │  cards (23 k PKM + 3 k OP + 85 k MTG)     │          │
│  │  card_images (rows + on-disk paths)       │          │
│  │  card_embeddings (HNSW table, 0 rows)     │          │
│  └───────────────────────────────────────────┘          │
│                                                         │
│  ┌─ embeddings (DINOv2 sidecar, internal) ─┐            │
│  │  Used by embedCatalog.ts to populate    │            │
│  │  card_embeddings (never run yet)        │            │
│  └─────────────────────────────────────────┘            │
└─────────────────────────────────────────────────────────┘

Who hosts what

Component	Source code	ML data	Runtime hosting
`tcg-vision` container	Cloned from `SKN/tgc-app` at `docker build`	`faiss_index.bin` baked into the image (built offline by Tormknd)	Our VPS
DINOv2 weights	`facebook/dinov2-small` (HuggingFace)	Downloaded on first boot, cached in the `tcg-vision-torch-cache` volume	Our VPS
`api` Fastify, `postgres`, `nginx`, `embeddings`, `ingest`	`snag-tcg/tcg-collector-backend` (our repo)	`cards` / `card_images` / `card_embeddings` ingested by us	Our VPS

Runtime conclusion: zero outbound calls during a scan. All processing happens on our VPS.

Build-time conclusion: tcg-vision.Dockerfile clones Tormknd's repo at a pinned commit (TCG_APP_COMMIT). If his repo disappeared tomorrow, the existing image keeps working; we just lose the ability to rebuild it.

The real dependency — `faiss_index.bin`

This is the one artefact we don't currently know how to regenerate without significant work.

Contents: ~30 370 DINOv2 vectors (~26 000 Pokémon, ~4 000 One Piece, a few hundred Lorcana)
How it was made: Tormknd embedded his own image folders (pokemon_cards_fr_high_res/, onepiece_cards_en/, etc.) with DINOv2-small and saved the resulting FAISS index.
Recent-set coverage: limited to whatever Tormknd had when he built it. Sets released after that date aren't in the index → scan returns "no-match".
For Magic: either Tormknd adds Magic to his repo and rebuilds, OR we build it ourselves.

The second pipeline — entirely ours, dormant

The pgvector + DINOv2 architecture already exists in the backend but isn't active yet:

Element	State	Role
Migration `0004_card_embeddings.sql`	✅ Applied	`card_embeddings (card_id, embedding vector(384), model)` table + HNSW cosine index
`embeddings:` service (compose)	✅ Up & healthy	Python DINOv2 sidecar (~250 ms/inference on CPU)
`src/ingest/embedCatalog.ts` script	✅ Exists, never run	Walks every card with a downloaded image → embeds → upserts into `card_embeddings`
API `/scan` embedding branch (`src/api/src/scan/embeddings.ts`)	✅ Coded, dormant	Queries `embedding <=> $1` (pgvector ANN cosine)
`card_embeddings` table	🟠 0 rows	Not populated

To activate this path:

bash

cd tcg-collector-backend
docker compose --env-file .env -f docker/compose.yml run --rm \
  --entrypoint "node dist/embedCatalog.js" ingest
# ~80 minutes for ~25 000 cards (existing Pokémon + One Piece)

After that, card_embeddings contains every ingested card and the API can serve /scan at a quality comparable to tcg-vision — with no external dependency.

How to add a TCG (general recipe)

Multi-TCG architecture is in place since migration 0005_tcg_classifier.sql (sets.tcg / cards.tcg columns). For a new TCG:

Pick a data source — see poc-tcg-collector/doc/COMPETITOR_DATA_SOURCES.md (Scryfall for MTG, YGOPRODeck for Yu-Gi-Oh!, OPTCG API for One Piece, Lorcast for Lorcana, etc.)
Write the client + orchestrator — model on src/ingest/optcgapi.ts + src/ingest/onepiece.ts: ~150 lines
Wire into the dispatch in src/ingest/src/index.ts — new branch if (tcg === 'magic') calling the orchestrator
Run the ingest — INGEST_TCG=<name> docker compose ... run --rm ingest
Handle the ML index:
- Option A — ask Tormknd: he adds the TCG to tgc-app, rebuilds his index, we bump TCG_APP_COMMIT and rebuild our image. Zero code on our side.
- Option B — pgvector: run embedCatalog.ts after the ingest to populate card_embeddings, and use /scan (api side) instead of /scan-compat (tcg-vision side).
Smoke test from the RN app — a scan of a card from the new TCG should return name + image + metadata via /scan-compat (or /scan depending on the path chosen). See tcg-collector-backend/docs/INGEST.md §One Piece for the precedent.

Endpoints today

Endpoint	Container	Scan path	Match source	Status
`:8000/scan-compat`	tcg-vision	DINOv2 + FAISS (Tormknd)	`faiss_index.bin`	Live, RN app points here
`:8000/scan`	tcg-vision	DINOv2 + FAISS (Tormknd)	same	Live, not used by the app
`:8080/scan`	api Fastify	pHash (weak signal) + dormant pgvector branch	`card_images` (pHash) + `card_embeddings` (empty)	Coded, degraded quality until pgvector populated

Recommended transition strategy when we want to drop the Tormknd dependency:

Run embedCatalog.ts to fill card_embeddings (~80 min)
A/B compare quality between /scan (api) and /scan-compat (tcg-vision) on a few scans
If equal or better → bump BACKEND_SCAN_URL in poc-tcg-collector to :8080/scan
At that point the tcg-vision container can be stopped (and its torch-cache volume reclaimed, ~85 MB)

Card scan architecture ​

TL;DR ​

Diagram ​

Who hosts what ​

The real dependency — faiss_index.bin ​

The second pipeline — entirely ours, dormant ​

How to add a TCG (general recipe) ​

Endpoints today ​

See also ​