Documentation

Appearance

3. Mobile — 5th "Frame" cell in the fan-out ​

Published on 2026-05-12

Completion of Path #5 « Frame » on the mobile side. First time today the user sees a concrete change in the app.

Before / after ​

Before todayFrom today onwards
Capture1 photo1 photo
Backends queried4 (Lab / Compat / Tormknd / Combined)5 (+ Frame)
Cells displayed45
Client-side preprocessingnonedetectQuad + warpQuad for the Frame cell

How the Frame cell works ​

The Scanner screen still runs Promise.allSettled over all modes. The new bit: for mode 'frame', the map callback takes a detour before the network call.

captured.path  ──▶  detectQuad(uri)
                       │
                ┌──────┴──────┐
                ▼             ▼
              null         { corners, confidence, ts }
                │             │
                │             ▼
                │        warpQuad(uri, corners)
                │             │
                │      ┌──────┴──────┐
                │      ▼             ▼
                │    null          "file:///cache/scan-frame-warp-<ts>.jpg"
                │      │             │
                ▼      ▼             ▼
       { ok: false, reason: 'no-quad-client' }   scanViaBackend({ path: warped, … },
                                                                 { endpointMode: 'frame',
                                                                   url: /scan-frame })

Why no fallback to the raw photo ​

When detectAndWarp returns null (card not detected, or warp failed), we send nothing to /scan-frame. An empty cell labeled no-quad-client shows up instead.

Reason: falling back to the raw photo would defeat the clean A/B. We want to know exactly when the client warp helps matching and when it doesn't. A silently-fallback cell would bias the numbers.

The 4 other cells keep firing with the raw photo in parallel, so the user always gets a result (unless every backend is down — in which case the existing on-device fallback kicks in).

Implementation ​

New failure type ​

ts
export type BackendScanFailure = {
  ok: false;
  reason:
    | 'network'
    | 'timeout'
    | 'http-4xx'
    | 'http-5xx'
    | 'malformed-response'
    | 'aborted'
    | 'no-quad-client';  // ← new
  // …
};

Distinct from 'malformed-response' (server returned garbage) and from a successful ok: true, match: null (server saw the embedding and found nothing). All three cases have different fixes — keeping them distinct is valuable for the debug UI.

New orchestrator ​

src/scan/quadDetector/detectAndWarp.ts:

ts
export async function detectAndWarp(photoUri: string): Promise<string | null> {
  const detection = await detectQuad(photoUri);
  if (detection === null) return null;
  return warpQuad(photoUri, detection.corners);
}

No try/catch — both wrappers already absorb their own errors into null (defensive 3-layer pattern documented in PR 2a / 4a).

Order in the fan-out ​

ts
const SCAN_MODES: readonly ScanEndpointMode[] = [
  'lab', 'compat', 'tormknd', 'combined', 'frame',
];

The 4 "raw-photo" cells first, the "client-warp" cell last. The A/B is then visual: on the same capture, you read row by row what each backend made of the raw photo, then what /scan-frame made of the rectified crop.

TDD ​

  • 4 new tests for detectAndWarp (detectQuad null skip, warpQuad null bubble, happy path, payload forwarding).
  • backendConfig.test.ts extended to assert the URL + port of mode 'frame'.
  • scanDebugFormat.test.ts extended to assert the 'Frame' label.

335 / 335 jest, tsc --noEmit clean.

What you'll see on device ​

  1. Scan a card → 5 cells appear in the summary, ordered Lab → Compat → Tormknd → Combined → Frame.

  2. Successful Frame cell → similarity shown, matched card image alongside. Should be ≥ that of /scan-lab if the client crop is good.

  3. Failed Frame cell → "no-quad-client" label. Typical causes:

    • Card edge poorly detected (insufficient contrast)
    • Card too close to a photo frame edge
    • Camera shake / motion blur
    • Too oblique an angle (beyond ~25-30°)

    The 4 other cells keep working normally in all those cases.

  4. Scan Debug now shows 5 rows per capture, grouped by inputPhotoPath.

Metric to watch ​

The ratio "Frame matches the same card_id as Lab" on real field photos. Direct indicator of:

  • the quality of the client-side OpenCV detection (detectQuad)
  • the quality of the perspective warp (warpQuad)
  • DINOv2's robustness to crop variations

If the ratio is high, we can consider dropping the server-side bounds detection (detectCardBounds in _detect_quad_in_edges) on the other routes — non-trivial CPU savings on the backend.

Further reading ​