Sealed scanner lab

Status: backend lab path merged on 2026-05-14. Updated on 2026-05-20 with Tormknd-style booster parity, dedicated DINOv2-base sealed embeddings, YOLO/OpenCV preprocessing, optional booster-pack routing, OCR metadata filtering, wider internal candidate retrieval, and candidate source diagnostics. Updated on 2026-05-23 to pause SKN-owned sealed references for normal scans.

The sealed scanner lab is the first backend path dedicated to Pokémon sealed products: booster packs, booster displays, booster bundles, Elite Trainer Boxes, blisters, tins, mini-tins, collections, and decks.

It deliberately does not reuse the raw-card schema or the card scan crop pipeline. A sealed product is not a card-shaped object, and forcing it through the card perspective warp would discard useful packaging geometry.

The default sealed path still uses full-image retrieval so the lab can compare against the original baseline. The preprocess=sealed mode is now YOLO-first: the embeddings sidecar tries a trained sealed-product segmentation model when SEALED_YOLO_MODEL_PATH points to one, then falls back to the previous axis-aligned OpenCV foreground crop. Both paths run before DINOv2 and are useful for phone captures where the table, hand, shelf, or screen background dominates the image.

The deployed stack must be checked through the sidecar health payload: if no trained YOLO file is mounted at /models/yolo/booster_seg.pt, generic sealed scans can still trigger a defensive raw full-image embedding and return preprocessingFallback.reason = sealed-preprocessing-without-yolo. When YOLO is available, generic sealed scans still compare the crop result against a raw full-image embedding and only fall back when raw clearly wins or resolves an ambiguity; that fallback reports preprocessingFallback.reason = raw-score-beats-yolo-crop. For booster-only backend A/B tests, callers can still send type=booster_pack&preprocess=sealed; that path intentionally trusts the YOLO/OpenCV crop result to match Tormknd's tcg-app2 booster scanner.

The target architecture is now explicit: sealed uses a two-brains flow. The first brain isolates the physical package with YOLO when weights are available, or OpenCV as a fallback. The second brain identifies the isolated package by embedding it with DINOv2 and searching the sealed catalog. Normal mobile scans now search ptcg-assets by default; SKN-owned references are kept for explicit debug or benchmark requests. OCR is not the first stage; it is a conditional reranker and metadata filter over the visual top-K when debug/accuracy experiments request it.

The important delta versus the previous SKN implementation is model separation and booster geometry. Card scans stay on facebook/dinov2-small with 384-dimensional vectors. Sealed scans now have their own sealed-embeddings sidecar using facebook/dinov2-base / ViT-B/14 with 768-dimensional vectors stored in sealed_product_embeddings_vitb_lab. Sealed/raw embeddings use the same DINO input geometry as tcg-app2 (Resize(526) then CenterCrop(518)) so booster wrappers keep their aspect ratio before matching. This matches the core idea in Tormknd's tcg-app2 booster scanner, while keeping our storage in Postgres/pgvector instead of Qdrant.

Early phone tests after the 2026-05-18 deployment show a clear qualitative precision improvement with preprocess=sealed. The reason is simple: DINOv2 now compares mostly the sealed product instead of comparing the sealed product plus desk, hand, monitor, shelf, or other background pixels. The crop preserves the packaging aspect ratio, so it improves signal-to-noise without applying the card perspective warp that would damage boxes, blisters, tins, and booster packs. This observation is promising, but it remains a field-test signal until measured on the separate sealed-eval benchmark.

What exists now

The backend repository exposes:

• POST /scan-sealed-lab on the Fastify API;
• sealed_products, sealed_product_images, and sealed_product_embeddings_lab in Postgres;
• sealed_product_embeddings_vitb_lab for the dedicated 768-dimensional sealed index;
• ingestSealedProductsLab.js to ingest local ptcg-assets files;
• raw full-image DINOv2-base sealed embeddings through the dedicated sealed sidecar with POST /embed?mode=raw;
• optional YOLO-first sealed preprocessing with POST /scan-sealed-lab?preprocess=sealed, which calls the sidecar with POST /embed?mode=sealed;
• defensive raw comparison for generic sealed scans: OpenCV fallback crops always get checked against raw, while YOLO crops are kept unless raw clearly wins or resolves ambiguity;
• booster-pack crop-first matching when the query includes type=booster_pack, mirroring tcg-app2;
• debug preprocessing diagnostics with ?debug=1, including method, bounding box, crop dimensions, and a JPEG data URI of the crop;
• optional OCR reranking with ?ocr=1, using the crop when available and exposing sealedOcr, rerank, and per-candidate visual/text/combined scores;
• strong OCR metadata filters before visual ranking when safe, for example Base Set text narrows to set_id = base1 and French packaging text narrows to language = fr;
• query filters for product type and language, for example ?type=booster_pack&lang=en;
• source filtering: normal requests default to source=ptcg-assets; debug can explicitly request source=skn-owned or source=all;
• metadata-only responses while the source remains internal-only.

The endpoint returns the best match, similarity, runner-up, top-K diagnostics, product type, set id, language, source path, and rights status.

The 2026-05-20 diagnosis confirmed that /scan-sealed-lab is not limited to the ten SKN-owned bootstrap photos. The active lab index can contain the broad ptcg-assets corpus plus skn-owned references. The misleading part was the debug surface: candidates did not clearly expose whether they came from ptcg-assets or skn-owned, and reranking could only work on the small SQL top-K returned by the first vector order.

The 2026-05-23 follow-up paused SKN-owned references for normal sealed scans because those owned bootstrap images could dominate real results. Runtime requests now default to source=ptcg-assets, even if old skn-owned rows are still present in Postgres. source=skn-owned and source=all remain available for debug, controlled A/B tests, and benchmark runs that deliberately measure owned references.

The endpoint now fetches a wider internal candidate pool before OCR reranking, then returns a compact debug topK to the app. This lets OCR rescue candidates that are visually close but were not in the first ten neighbours, without making the mobile debug carousel huge. Candidate objects expose source and productType in addition to type, so Scan Debug can distinguish a real ptcg-assets neighbour from an owned bootstrap reference.

It intentionally returns productDetails.image = null for ptcg-assets rows. That repository has no declared license, so SKN must not expose those images publicly or copy them into a public CDN/app bundle.

The endpoint also refuses near-tie neighbours. When the top candidate is less than 0.015 combined-score points ahead of the runner-up, it returns match: null with reason: ambiguous-neighbor while keeping runnerUp and topK diagnostics. This avoids presenting visually close sealed variants as certain matches. OCR can now resolve some of these near-ties when product text clearly supports one of the visual candidates.

OCR set-name filtering also maps Wizards-era names to the catalog IDs used by ptcg-assets: Base Set / Edition Base -> base1, Jungle -> base2, Fossil / Fossile -> base3, and Team Rocket -> base5. This matters because filtering on literal fossil or jungle would remove the correct ptcg-assets rows before visual ranking.

Why raw visual retrieval first

For sealed products, the fastest accurate first pass is visual retrieval:

1. encode the whole uploaded package image with DINOv2;
2. search nearest neighbours in sealed_product_embeddings_vitb_lab with pgvector HNSW;
3. use optional filters such as product type or language when the UI already knows them.

OCR is a second-stage reranker and metadata filter, not the first blocking step. Packaging text is frequently distorted by plastic wrap, glare, localization, angled phone photos, and small foil typography. OCR is still useful, but mostly after visual retrieval has narrowed the candidate set or when a strong keyword can safely reduce the candidate space.

The current ambiguity guard is a false-positive control, not the final accuracy engine. It turns close calls into explicit ambiguities; the current accuracy path resolves some of them with crop-aware OCR, while the next stronger steps are trained YOLO weights, product-type/language filters, and larger field benchmarks.

The current app Sealed mode appends debug=1&ocr=1 and keeps the upload raw by default. This is deliberate: one generic Sealed UI must still match ETBs, displays, blisters, tins, collection boxes, and booster packs, and a booster-only crop can incorrectly lock onto a visible pack inside a larger product. The app only appends preprocess=sealed&typeHint=booster_pack when the user selects a booster-style frame. Backend A/B tests can still add type=booster_pack&preprocess=sealed when they intentionally want to test the booster-only path with a hard product-type filter.

This is the main difference from Tormknd's booster demo path: his flow is booster-specific, while the SKN mobile Sealed mode is currently a generic sealed-product path. Generic matching is broader and therefore harder. If the test objective is only a booster pack, the backend supports the same booster-only constraint through type=booster_pack&preprocess=sealed, while the mobile booster-style frame sends the softer typeHint=booster_pack&preprocess=sealed; if the product objective is all sealed types, the scanner needs either subtype controls or detector-backed routing.

The backend also materializes the filtered sealed candidate set before sorting by vector distance. This is intentionally more exact than a filtered ANN-only query: the sealed catalog is small, and filtered HNSW scans can otherwise return an empty top-K even when the requested family, such as booster_pack, has hundreds of embeddings.

Operationally, YOLO is not the final product identifier. It detects and crops or masks the sealed object. DINOv2 then embeds that crop, and pgvector searches the nearest sealed references. If the YOLO model is absent, unloadable, or below threshold, the sidecar keeps scanning with the OpenCV fallback rather than failing the request. Explicit type=booster_pack A/B scans trust that crop-first result; generic sealed scans compare against a raw full-image embedding as a guardrail because the current bootstrap YOLO model is booster-only and can crop a visible booster inside a broader sealed product.

The sealed ingest also compresses oversized reference uploads before they reach the embedding sidecar. Source images and provenance remain unchanged, but very large PNGs are sent as bounded JPEG payloads to avoid 413 file-too-large holes in the 768-d reference table.

YOLO bootstrap from tcg-app2

tcg-app2 already contains a one-class YOLO segmentation dataset for boosters. SKN can use that dataset as a local bootstrap without modifying the reference repository and without committing trained weights.

From tcg-collector-backend:

scripts/train-sealed-yolo.sh
docker compose --env-file .env -f docker/compose.yml restart sealed-embeddings embeddings

The script expects ../tcg-app2/dataset, trains inside the already-built sealed-embeddings Docker image, and installs the gitignored artifact at:

tcg-collector-backend/models/yolo/booster_seg.pt

After restart, a sealed debug request should report preprocessing.method = yolo-segmentation for a booster photo. This closes the gap between “the backend supports YOLO” and “the running stack is actually using YOLO”.

Mobile status

The mobile app is wired to this path for the manual sealed test mode.

In poc-tcg-collector, the Scanner item-type selector now controls the backend plan:

• Card and Slab keep the five diagnostic card endpoints: lab, compat, tormknd, combined, and frame;
• Sealed hides the TCG card filter row and calls only POST /scan-sealed-lab.

As of poc-tcg-collector#194, the default sealed frame is Auto, not Pack. This is intentional. A generic sealed scan must not send typeHint=booster_pack by default, because that biases imported ETBs, displays, bundles, tins, blisters, and collection boxes toward booster-pack neighbours before the visual match is evaluated.

Imported sealed images are also neutral: the app records whether the input came from the camera or the image library, and library imports do not inherit the active frame preset's product-type hint. Manual camera tests can still select the Pack frame to send the soft booster_pack hint. This preserves the booster A/B path without poisoning imported cross-type sealed tests.

The Scanner test panel and the separate Scan Debug screen both preserve sealed diagnostics. Scan Debug labels those rows as Pokémon sealed, shows product type / language / rights metadata, and can display input -> crop -> match/candidate. The crop comes from private preprocessing debug data, so a bad result can be diagnosed as crop failure versus retrieval/rerank failure.

Candidate cards in the debug surfaces now include candidate source and productType. When a result is wrong, this makes the failure review concrete: wrong ptcg-assets product family, wrong language/SKU, true visual ambiguity, or deliberate debug runs dominated by skn-owned bootstrap references.

This is still a lab surface. The app does not auto-detect sealed packaging yet. It does not display source images from ptcg-assets because those rows remain internal-only; it can display private debug images for skn-owned bootstrap references through /debug/sealed-owned/....

Scan Debug case archive

Internal lab builds can now save Scan Debug captures to the backend with POST /scan-debug-cases. This is not public telemetry and it is not automatic model training. It is a controlled case archive that gives the team the raw material needed to review failures, label them, and later promote useful examples into evaluation or training datasets.

Each saved case contains:

• input.jpg: the original photo or imported image sent by the app;
• metadata.json: item type, scan mode, local Scan Debug id, backend result, top-K candidates, scores, failure reason, and timings when present;
• crop.jpg: optional private preprocessing crop extracted from debug metadata when the backend returned one.

The endpoint is protected by the backend write token (API_WRITE_TOKEN) because it stores user-supplied images. The mobile upload is opt-in through the gitignored local config file in the POC app. If the local config is missing, Scan Debug still works locally and no upload is attempted.

These archived cases are the feedback loop:

1. scan or import a real item;
2. let the app save the backend result and diagnostics;
3. review the case later and mark the expected product, failure type, and useful notes;
4. promote labelled cases into sealed-eval or another benchmark only when they are owned or explicitly permitted;
5. use the benchmark to decide whether the next fix is better crop detection, OCR/rerank, metadata filtering, more references, or a trained model.

Use the backend command after a test session:

cd tcg-collector-backend/src/api
npm run scan-debug:review -- --markdown /tmp/scan-debug-review.md

For visual review, open the internal Cardex backend app and use the scan debug section. It shows the archived input and crop next to the returned match, reason, review status, and top-K candidate IDs.

Internal asset boundary

ptcg-assets is useful for proving the sealed scanner because it contains a broad packaging corpus and a useful file taxonomy. It is not production-safe by itself.

The lab can also index a small skn-owned source: ten French sealed photos owned by SKN and stored in tcg-collector-backend/src/api/tests/fixtures/sealed-samples. These rows use stable skn-owned:* product IDs and rights_status = 'approved'. They are bootstrap references, not a complete production catalog, and they are paused by default for normal mobile scans.

Current backend guardrails:

• every row from ptcg-assets is marked internal_only;
• every row from skn-owned is tied to an owned fixture and marked approved, but normal /scan-sealed-lab requests exclude it by default;
• the ingest stores source provenance instead of public image URLs;
• the API keeps productDetails.image null and exposes source images only as private debug candidate URLs;
• the Docker mounts are read-only;
• the sealed tables are separate from card tables and card embeddings.

Production requires one of:

• explicit permission from the asset owner;
• a compatible license;
• a paid/provider source with terms that allow the intended usage;
• user-owned uploaded images with consent and retention controls;
• a curated allowlist of assets whose rights are documented.

Runbook

From tcg-collector-backend:

docker compose --env-file .env -f docker/compose.yml up -d sealed-embeddings
docker compose --env-file .env -f docker/compose.yml run --rm \
  --entrypoint "node dist/ingestSealedProductsLab.js" ingest

The compose default keeps SKN-owned references paused during ingest:

SEALED_INCLUDE_SKN_REFERENCES=false

If a debug session intentionally needs owned bootstrap references, enable them explicitly and query them explicitly:

# runtime query examples
/scan-sealed-lab?source=skn-owned
/scan-sealed-lab?source=all

Useful constrained run:

SEALED_ASSET_SETS=tcgpa2b \
SEALED_ASSET_TYPES=booster_pack \
SEALED_ASSET_LIMIT=1 \
docker compose --env-file .env -f docker/compose.yml run --rm \
  --entrypoint "node dist/ingestSealedProductsLab.js" ingest

Smoke the endpoint with a JPEG:

curl -s \
  -F "photo=@../ptcg-assets/tcgpa2b/packshots/booster.jpg;type=image/jpeg" \
  "http://localhost:8080/scan-sealed-lab?type=booster_pack" | jq .

Run the sealed benchmark against labeled samples and optional inbox photos:

cd src/api
npm run benchmark:sealed -- --include-inbox --json /tmp/sealed-benchmark.json

Compare the raw full-image path with the sealed foreground crop:

npm run benchmark:sealed -- --json /tmp/sealed-raw.json
npm run benchmark:sealed -- --preprocess sealed --json /tmp/sealed-crop.json

The benchmark reads tests/fixtures/sealed-samples/manifest.json as the labeled source of truth and can also scan tests/fixtures/sealed-samples/images/inbox/ as unlabeled diagnostics. Inbox samples are useful for candidate review, but they are excluded from accuracy until promoted into the manifest.

• exactAccuracy counts labeled samples where the top-1 match is the expected product ID.
• topKAccuracy also counts labeled samples where the expected product appears in topK after an ambiguous-neighbor response.
• expectedProductId should be preferred once the catalog row exists; acceptableProductIds can hold legitimate variants during catalog cleanup.

Current bootstrap result after ingesting the skn-owned source on 2026-05-16:

• 10/10 exact matches without filters;
• 10/10 exact matches with --use-label-filters;
• this validates the French sealed ingest/query/debug path, but it should not be reported as market accuracy because the benchmark photos are also the bootstrap reference images.

As of 2026-05-23, this bootstrap path should be treated as opt-in only. Use --source skn-owned to validate owned reference wiring, or --source all for an explicit comparison. Normal mobile scans should stay on the default ptcg-assets source until the field-evaluation dataset proves that owned references improve generalization instead of dominating results.

Field-evaluation dataset

Unseen phone photos now have a separate evaluation home in the backend:

tcg-collector-backend/src/api/tests/fixtures/sealed-eval/
  manifest.json
  images/inbox/
  images/labeled/

This folder is intentionally separate from sealed-samples. sealed-samples contains SKN-owned bootstrap reference images that may be indexed as skn-owned sealed products. sealed-eval contains real field photos that must stay out of reference ingest, even when they are labelled. The backend test suite rejects evaluationOnly: true rows if someone tries to ingest them as SKN-owned references.

Use images/inbox/ for owned photos whose label is still unknown. Use images/labeled/ plus manifest.json for photos with expected metadata. Every labelled row must include:

• evaluationOnly: true;
• a relative JPEG file under images/;
• expectedProductName;
• expectedProductType;
• captureSource;
• rights set to owned_by_skn or explicit_permission;
• expectedProductId when the catalog row exists, otherwise metadata-only evaluation is allowed.

Run the field evaluation from src/api/ with the backend stack running:

npm run benchmark:sealed:eval -- --json /tmp/sealed-field-eval.json

The command includes inbox photos for diagnostic review. Empty dataset bootstraps can be checked with:

npm run benchmark:sealed:eval -- --allow-empty

Each failure should be classified before changing thresholds:

• expected product in top-K: add OCR or hybrid reranking;
• wrong product type dominates: add product-type filter or detector crop;
• expected product absent: add licensed references or improve crop;
• noisy phone photo fails broadly: prioritize sealed detection/cropping.

The benchmark runner now writes this classification directly in the console and JSON report:

diagnosis	nextAction	Meaning
expected-in-topk	add-ocr-reranking	The correct product is nearby, but the final ranking needs text or metadata evidence.
wrong-product-type	add-type-filter-or-detector-crop	Retrieval is comparing against the wrong sealed family, such as booster pack versus blister.
wrong-language	add-language-evidence-or-reference	Artwork is close, but the locale/SKU evidence is wrong.
expected-absent	add-reference-or-improve-crop	The expected product is not retrieved; add references or improve the crop.
unlabeled-review	label-sample	The sample can be reviewed manually, but it does not count toward accuracy yet.

For larger runs, the JSON summary also includes summary.diagnoses and summary.nextActions, and the console prints diagnoses: / next actions: aggregate counts.

Next steps

1. Fill sealed-eval with unseen owned phone photos and run it separately from the bootstrap benchmark.
2. Keep SKN-owned references opt-in until field evaluation shows they help unseen phone photos.
3. Add sealed subtype controls or detector-backed routing for non-booster products such as ETB / display / tin / blister.
4. Add a camera UI frame that matches sealed packaging without assuming card proportions.
5. Add reviewer labels and correction feedback to the saved Scan Debug case archive.
6. Train or import production-quality YOLO sealed weights at models/yolo/booster_seg.pt.
7. Benchmark whether sealed should stay on DINOv2-base + pgvector or move the vector store to Qdrant later.
8. Define a production-safe asset and image display policy before public launch.

Related pages

• Asset licensing and source risk register
• Data sources and pricing strategy
• Scan UI/UX competitive review
• Product engine architecture