RVL-CDIP OpenRouter multi-model eval (PoC)

Follow-on to rvl_cdip_recreation_sampling.ipynb.

Loads the recreation experiment set (recreation_samples.jsonl, typically 1040 rows = 65 × 16 classes), materializes page images (selective tar extract when the archive is present), optionally OCRs them, then queries OpenRouter models to:

  1. Classify into the 16 RVL-CDIP labels (gold labels available)
  2. Extract / annotate structured insurance-style fields (proxy metrics — no gold fields)

Modality: vision-first (page image → vision models); OCR text → text models.

Related: RVL-CDIP SQL Index · CLI python -m src.rvl_cdip --help

0. Prerequisites

python scripts/setup_env.py          # OPENROUTER_API_KEY in .env
python -m src.rvl_cdip build         # labels + SQLite (if not already)
pip install -e ".[ocr]"             # only if using the text/OCR path

# Image archive (~38 GB) — preview then dual opt-in (writes ONLY under .venv/rvl_cdip):
python -m src.rvl_cdip download-images --preflight
python -m src.rvl_cdip download-images \
  --i-understand-large-download --confirm-writes-under-venv

Defaults below keep DRY_RUN=True so this notebook is safe for docs/CI.

from __future__ import annotations

import json
from pathlib import Path

import pandas as pd
from IPython.display import Markdown, display

REPO_ROOT = Path.cwd()
if not (REPO_ROOT / "pyproject.toml").exists():
    for cand in [REPO_ROOT.parent, *REPO_ROOT.parents]:
        if (cand / "pyproject.toml").exists():
            REPO_ROOT = cand
            break

import sys

if str(REPO_ROOT) not in sys.path:
    sys.path.insert(0, str(REPO_ROOT))

from src.rvl_cdip.download import (
    download_images,
    format_image_download_preflight,
    image_download_preflight,
)
from src.rvl_cdip.openrouter_eval import (
    DEFAULT_TEXT_MODELS,
    DEFAULT_VISION_MODELS,
    load_samples,
    run_eval,
)
from src.rvl_cdip.paths import LABEL_NAMES, archive_path
from src.rvl_cdip.sample_images import materialize_samples, summarize_materialize
from src.rvl_cdip.store import RvlCdipStore
from src.utils.config import Config

SAMPLES_PATH = REPO_ROOT / "data/notebook_demo/rvl_cdip_recreation/recreation_samples.jsonl"
OUT_DIR = REPO_ROOT / "data/notebook_demo/rvl_cdip_openrouter_eval"

# --- knobs -------------------------------------------------------------------
DRY_RUN = True          # False → live OpenRouter calls (needs OPENROUTER_API_KEY)
MAX_PER_CLASS = 65      # full PoC = 65 → 1040 docs; use 2–5 for cheap smoke
RUN_OCR = True          # needed for text modality
MODALITIES = ("vision", "text")
VISION_MODELS = list(DEFAULT_VISION_MODELS)
TEXT_MODELS = list(DEFAULT_TEXT_MODELS)

cfg = Config.load()
has_key = bool(cfg.openrouter_api_key)
display(
    Markdown(
        f"Repo `{REPO_ROOT}`  \n"
        f"`OPENROUTER_API_KEY` set: **{has_key}**  ·  `DRY_RUN={DRY_RUN}`  ·  "
        f"`MAX_PER_CLASS={MAX_PER_CLASS}`  ·  archive present: **{archive_path().is_file()}**"
    )
)

Repo /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer
OPENROUTER_API_KEY set: True · DRY_RUN=True · MAX_PER_CLASS=65 · archive present: False

0b. Optional: download image archive into .venv/rvl_cdip only

Hub cache + rvl-cdip.tar.gz write only under .venv/rvl_cdip/ (never data/ or ~/.cache). Set both flags below to True after reviewing the preflight block, then run the cell. Prefer CLI for long downloads:

python -m src.rvl_cdip download-images --preflight
python -m src.rvl_cdip download-images \
  --i-understand-large-download --confirm-writes-under-venv
# interactive phrase prompt instead of the second flag:
# python -m src.rvl_cdip download-images \
#   --i-understand-large-download --interactive-confirm
I_UNDERSTAND_LARGE_DOWNLOAD = False   # ~38 GB archive
CONFIRM_WRITES_UNDER_VENV = False     # must acknowledge .venv/rvl_cdip-only writes
FORCE_REDOWNLOAD = False

preflight = image_download_preflight(force=FORCE_REDOWNLOAD)
print(format_image_download_preflight(preflight))
display(preflight)

if I_UNDERSTAND_LARGE_DOWNLOAD and CONFIRM_WRITES_UNDER_VENV:
    if not preflight["enough_free_space"] and not preflight["archive_already_present"]:
        raise RuntimeError(
            f"Need ≥ {preflight['min_free_gb']} GB free; have {preflight['free_gb']} GB."
        )
    result = download_images(
        force=FORCE_REDOWNLOAD,
        i_understand_large_download=True,
        confirm_writes_under_venv=True,
    )
    updated = RvlCdipStore().refresh_image_paths()
    display(
        {
            "local_path": str(result.local_path),
            "bytes": result.bytes,
            "skipped": result.skipped,
            "image_abspath_updated": updated,
            "writes_only_under": result.detail.get("writes_only_under"),
        }
    )
else:
    display(
        Markdown(
            "> **Download not started.** Set `I_UNDERSTAND_LARGE_DOWNLOAD = True` and "
            "`CONFIRM_WRITES_UNDER_VENV = True` after reviewing the preflight "
            "(writes only under `.venv/rvl_cdip`)."
        )
    )
RVL-CDIP image archive download — preflight

CONFIRMATION: all Hub cache + archive writes stay under:
  /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip
  (not data/, not ~/.cache/huggingface)

Remote:     aharley/rvl_cdip/data/rvl-cdip.tar.gz
Archive →   /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/source/rvl-cdip.tar.gz
HF home →   /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/hf_home
Images dir: /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/source/images

Archive size (est.): 36.1 GB
Free space now:      56.73 GB (need ≥ 45.0 GB)
Enough free space:   True
Already present:     False

Ack phrase: "writes only under .venv/rvl_cdip"
{'dataset_id': 'aharley/rvl_cdip',
 'remote_ref': 'aharley/rvl_cdip/data/rvl-cdip.tar.gz',
 'writes_only_under': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip',
 'archive_path': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/source/rvl-cdip.tar.gz',
 'source_dir': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/source',
 'images_dir': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/source/images',
 'hf_home': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/hf_home',
 'archive_estimate_gb': 36.1,
 'min_free_gb': 45.0,
 'free_gb': 56.73,
 'free_bytes': 60917178368,
 'enough_free_space': True,
 'archive_already_present': False,
 'confirmation_phrase': 'writes only under .venv/rvl_cdip'}

Download not started. Set I_UNDERSTAND_LARGE_DOWNLOAD = True and CONFIRM_WRITES_UNDER_VENV = True after reviewing the preflight (writes only under .venv/rvl_cdip).

1. Load recreation samples

samples = load_samples(SAMPLES_PATH, max_per_class=MAX_PER_CLASS)
df = pd.DataFrame(samples)
display(Markdown(f"Loaded **{len(samples)}** samples from `{SAMPLES_PATH.relative_to(REPO_ROOT)}`"))
display(df.groupby(["label_id", "label"], as_index=False).size().head(20))
df.head(3)

Loaded 1040 samples from data/notebook_demo/rvl_cdip_recreation/recreation_samples.jsonl

label_id label size
0 0 letter 65
1 1 form 65
2 2 email 65
3 3 handwritten 65
4 4 advertisement 65
5 5 scientific report 65
6 6 scientific publication 65
7 7 specification 65
8 8 file folder 65
9 9 news article 65
10 10 budget 65
11 11 invoice 65
12 12 presentation 65
13 13 questionnaire 65
14 14 resume 65
15 15 memo 65
document_id split label_id label image_relpath image_abspath source_line recreation_seed n_per_class split_filter
0 test:imagesz/z/p/v/zpv93d00/508330972_50833097... test 0 letter imagesz/z/p/v/zpv93d00/508330972_508330974.tif None 13284 42 65 None
1 train:imagese/e/u/v/euv68e00/2001213893.tif train 0 letter imagese/e/u/v/euv68e00/2001213893.tif None 57555 42 65 None
2 train:imagesp/p/s/e/pse16c00/2071710123.tif train 0 letter imagesp/p/s/e/pse16c00/2071710123.tif None 13386 42 65 None

2. Materialize images (+ optional OCR)

Resolution order: existing image_abspath → .venv/rvl_cdip/source/images/{relpath} → extract that member from rvl-cdip.tar.gz when the archive is present.

mats = materialize_samples(samples, run_ocr=RUN_OCR and ("text" in MODALITIES))
mat_summary = summarize_materialize(mats)
display(Markdown("### Materialize summary"))
display(mat_summary)
mat_df = pd.DataFrame([m.to_dict() for m in mats])
display(mat_df.head(5))
if mat_summary["n_with_image"] == 0 and "vision" in MODALITIES and not DRY_RUN:
    display(
        Markdown(
            "> **Vision path blocked:** no images on disk and archive missing. "
            "Use section **0b** or\n"
            "`python -m src.rvl_cdip download-images --i-understand-large-download "
            "--confirm-writes-under-venv`, then re-run materialize "
            "(selective extract of the sample paths into `.venv/rvl_cdip/source/images/`)."
        )
    )

Materialize summary

{'n_samples': 1040,
 'n_with_image': 0,
 'n_extracted': 0,
 'n_with_ocr': 0,
 'n_errors': 1040,
 'archive_present': False,
 'images_dir': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/source/images'}
document_id image_relpath image_path extracted ocr_path ocr_chars error
0 test:imagesz/z/p/v/zpv93d00/508330972_50833097... imagesz/z/p/v/zpv93d00/508330972_508330974.tif None False None 0 image not on disk and archive missing (/Users/...
1 train:imagese/e/u/v/euv68e00/2001213893.tif imagese/e/u/v/euv68e00/2001213893.tif None False None 0 image not on disk and archive missing (/Users/...
2 train:imagesp/p/s/e/pse16c00/2071710123.tif imagesp/p/s/e/pse16c00/2071710123.tif None False None 0 image not on disk and archive missing (/Users/...
3 validation:imagesx/x/g/p/xgp60d00/522473908+-3... imagesx/x/g/p/xgp60d00/522473908+-3908.tif None False None 0 image not on disk and archive missing (/Users/...
4 train:imagesp/p/a/z/paz81e00/01353637_01353639... imagesp/p/a/z/paz81e00/01353637_01353639.tif None False None 0 image not on disk and archive missing (/Users/...

3. Run OpenRouter eval (classify + extract/annotate)

Two calls per document × model × modality. Predictions append to predictions.jsonl with resume/cache on (document_id, model_id, task, modality).

if not DRY_RUN and not has_key:
    raise RuntimeError("Set OPENROUTER_API_KEY (or keep DRY_RUN=True)")

manifest = run_eval(
    samples,
    materialize=mats,
    vision_models=VISION_MODELS,
    text_models=TEXT_MODELS,
    modalities=MODALITIES,
    out_dir=OUT_DIR,
    dry_run=DRY_RUN,
    cfg=cfg,
    resume=True,
)
display(Markdown(f"Wrote artifacts under `{OUT_DIR.relative_to(REPO_ROOT)}`"))
display(manifest)

Wrote artifacts under data/notebook_demo/rvl_cdip_openrouter_eval

{'experiment': 'rvl_cdip_openrouter_eval',
 'created_at': '2026-07-23T05:12:14.069919+00:00',
 'n_samples': 1040,
 'vision_models': ['openai/gpt-4o-mini', 'google/gemini-2.0-flash-001'],
 'text_models': ['openai/gpt-4o-mini',
  'meta-llama/llama-3.2-3b-instruct:free'],
 'tasks': ['classify', 'extract_annotate'],
 'modalities': ['vision', 'text'],
 'dry_run': True,
 'predictions_path': 'data/notebook_demo/rvl_cdip_openrouter_eval/predictions.jsonl',
 'materialize': {'n_samples': 1040,
  'n_with_image': 0,
  'n_extracted': 0,
  'n_with_ocr': 0,
  'n_errors': 1040,
  'archive_present': False,
  'images_dir': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/.venv/rvl_cdip/source/images'},
 'classification': {'per_model': [{'model_id': 'google/gemini-2.0-flash-001',
    'modality': 'vision',
    'n': 1040,
    'accuracy': 1.0,
    'macro_recall': 1.0,
    'per_class_recall': {'letter': 1.0,
     'form': 1.0,
     'email': 1.0,
     'handwritten': 1.0,
     'advertisement': 1.0,
     'scientific report': 1.0,
     'scientific publication': 1.0,
     'specification': 1.0,
     'file folder': 1.0,
     'news article': 1.0,
     'budget': 1.0,
     'invoice': 1.0,
     'presentation': 1.0,
     'questionnaire': 1.0,
     'resume': 1.0,
     'memo': 1.0},
    'error_rate': 0.0,
    'avg_latency_seconds': 0.0,
    'total_input_tokens': 0,
    'total_output_tokens': 0},
   {'model_id': 'meta-llama/llama-3.2-3b-instruct:free',
    'modality': 'text',
    'n': 1040,
    'accuracy': 1.0,
    'macro_recall': 1.0,
    'per_class_recall': {'letter': 1.0,
     'form': 1.0,
     'email': 1.0,
     'handwritten': 1.0,
     'advertisement': 1.0,
     'scientific report': 1.0,
     'scientific publication': 1.0,
     'specification': 1.0,
     'file folder': 1.0,
     'news article': 1.0,
     'budget': 1.0,
     'invoice': 1.0,
     'presentation': 1.0,
     'questionnaire': 1.0,
     'resume': 1.0,
     'memo': 1.0},
    'error_rate': 0.0,
    'avg_latency_seconds': 0.0,
    'total_input_tokens': 0,
    'total_output_tokens': 0},
   {'model_id': 'openai/gpt-4o-mini',
    'modality': 'text',
    'n': 1040,
    'accuracy': 1.0,
    'macro_recall': 1.0,
    'per_class_recall': {'letter': 1.0,
     'form': 1.0,
     'email': 1.0,
     'handwritten': 1.0,
     'advertisement': 1.0,
     'scientific report': 1.0,
     'scientific publication': 1.0,
     'specification': 1.0,
     'file folder': 1.0,
     'news article': 1.0,
     'budget': 1.0,
     'invoice': 1.0,
     'presentation': 1.0,
     'questionnaire': 1.0,
     'resume': 1.0,
     'memo': 1.0},
    'error_rate': 0.0,
    'avg_latency_seconds': 0.0,
    'total_input_tokens': 0,
    'total_output_tokens': 0},
   {'model_id': 'openai/gpt-4o-mini',
    'modality': 'vision',
    'n': 1040,
    'accuracy': 1.0,
    'macro_recall': 1.0,
    'per_class_recall': {'letter': 1.0,
     'form': 1.0,
     'email': 1.0,
     'handwritten': 1.0,
     'advertisement': 1.0,
     'scientific report': 1.0,
     'scientific publication': 1.0,
     'specification': 1.0,
     'file folder': 1.0,
     'news article': 1.0,
     'budget': 1.0,
     'invoice': 1.0,
     'presentation': 1.0,
     'questionnaire': 1.0,
     'resume': 1.0,
     'memo': 1.0},
    'error_rate': 0.0,
    'avg_latency_seconds': 0.0,
    'total_input_tokens': 0,
    'total_output_tokens': 0}]},
 'extraction': {'per_model': [{'model_id': 'google/gemini-2.0-flash-001',
    'modality': 'vision',
    'n': 1040,
    'valid_json_rate': 1.0,
    'mean_field_fill_rate': 0.4,
    'field_fill_rates': {'document_type_guess': 1.0,
     'title_or_subject': 1.0,
     'parties': 0.0,
     'dates': 0.0,
     'identifiers': 0.0,
     'amounts': 0.0,
     'addresses_or_locations': 0.0,
     'key_value_fields': 0.0,
     'summary': 1.0,
     'extraction_notes': 1.0},
    'error_rate': 0.0},
   {'model_id': 'meta-llama/llama-3.2-3b-instruct:free',
    'modality': 'text',
    'n': 1040,
    'valid_json_rate': 1.0,
    'mean_field_fill_rate': 0.4,
    'field_fill_rates': {'document_type_guess': 1.0,
     'title_or_subject': 1.0,
     'parties': 0.0,
     'dates': 0.0,
     'identifiers': 0.0,
     'amounts': 0.0,
     'addresses_or_locations': 0.0,
     'key_value_fields': 0.0,
     'summary': 1.0,
     'extraction_notes': 1.0},
    'error_rate': 0.0},
   {'model_id': 'openai/gpt-4o-mini',
    'modality': 'text',
    'n': 1040,
    'valid_json_rate': 1.0,
    'mean_field_fill_rate': 0.4,
    'field_fill_rates': {'document_type_guess': 1.0,
     'title_or_subject': 1.0,
     'parties': 0.0,
     'dates': 0.0,
     'identifiers': 0.0,
     'amounts': 0.0,
     'addresses_or_locations': 0.0,
     'key_value_fields': 0.0,
     'summary': 1.0,
     'extraction_notes': 1.0},
    'error_rate': 0.0},
   {'model_id': 'openai/gpt-4o-mini',
    'modality': 'vision',
    'n': 1040,
    'valid_json_rate': 1.0,
    'mean_field_fill_rate': 0.4,
    'field_fill_rates': {'document_type_guess': 1.0,
     'title_or_subject': 1.0,
     'parties': 0.0,
     'dates': 0.0,
     'identifiers': 0.0,
     'amounts': 0.0,
     'addresses_or_locations': 0.0,
     'key_value_fields': 0.0,
     'summary': 1.0,
     'extraction_notes': 1.0},
    'error_rate': 0.0}],
  'pairwise_agreement': {'n_pairs': 6,
   'pairs': [{'model_a': 'google/gemini-2.0-flash-001|vision',
     'model_b': 'meta-llama/llama-3.2-3b-instruct:free|text',
     'n_compared_fields': 3120,
     'agreement_rate': 1.0},
    {'model_a': 'google/gemini-2.0-flash-001|vision',
     'model_b': 'openai/gpt-4o-mini|text',
     'n_compared_fields': 3120,
     'agreement_rate': 1.0},
    {'model_a': 'google/gemini-2.0-flash-001|vision',
     'model_b': 'openai/gpt-4o-mini|vision',
     'n_compared_fields': 3120,
     'agreement_rate': 1.0},
    {'model_a': 'meta-llama/llama-3.2-3b-instruct:free|text',
     'model_b': 'openai/gpt-4o-mini|text',
     'n_compared_fields': 3120,
     'agreement_rate': 1.0},
    {'model_a': 'meta-llama/llama-3.2-3b-instruct:free|text',
     'model_b': 'openai/gpt-4o-mini|vision',
     'n_compared_fields': 3120,
     'agreement_rate': 1.0},
    {'model_a': 'openai/gpt-4o-mini|text',
     'model_b': 'openai/gpt-4o-mini|vision',
     'n_compared_fields': 3120,
     'agreement_rate': 1.0}]}}}

4. Scoreboards

  • Classification: accuracy + macro recall vs gold label
  • Extraction: valid-JSON rate, field fill-rates, pairwise agreement (no gold fields)
cls_path = OUT_DIR / "summary_classification.json"
ext_path = OUT_DIR / "summary_extraction.json"
cls = json.loads(cls_path.read_text()) if cls_path.exists() else {}
ext = json.loads(ext_path.read_text()) if ext_path.exists() else {}

cls_df = pd.DataFrame(cls.get("per_model") or [])
ext_df = pd.DataFrame(ext.get("per_model") or [])
display(Markdown("### Classification"))
display(cls_df)
display(Markdown("### Extraction proxies"))
display(ext_df)
if ext.get("pairwise_agreement"):
    display(Markdown("### Pairwise agreement"))
    display(pd.DataFrame(ext["pairwise_agreement"].get("pairs") or []))

Classification

model_id modality n accuracy macro_recall per_class_recall error_rate avg_latency_seconds total_input_tokens total_output_tokens
0 google/gemini-2.0-flash-001 vision 1040 1.0 1.0 {'letter': 1.0, 'form': 1.0, 'email': 1.0, 'ha... 0.0 0.0 0 0
1 meta-llama/llama-3.2-3b-instruct:free text 1040 1.0 1.0 {'letter': 1.0, 'form': 1.0, 'email': 1.0, 'ha... 0.0 0.0 0 0
2 openai/gpt-4o-mini text 1040 1.0 1.0 {'letter': 1.0, 'form': 1.0, 'email': 1.0, 'ha... 0.0 0.0 0 0
3 openai/gpt-4o-mini vision 1040 1.0 1.0 {'letter': 1.0, 'form': 1.0, 'email': 1.0, 'ha... 0.0 0.0 0 0

Extraction proxies

model_id modality n valid_json_rate mean_field_fill_rate field_fill_rates error_rate
0 google/gemini-2.0-flash-001 vision 1040 1.0 0.4 {'document_type_guess': 1.0, 'title_or_subject... 0.0
1 meta-llama/llama-3.2-3b-instruct:free text 1040 1.0 0.4 {'document_type_guess': 1.0, 'title_or_subject... 0.0
2 openai/gpt-4o-mini text 1040 1.0 0.4 {'document_type_guess': 1.0, 'title_or_subject... 0.0
3 openai/gpt-4o-mini vision 1040 1.0 0.4 {'document_type_guess': 1.0, 'title_or_subject... 0.0

Pairwise agreement

model_a model_b n_compared_fields agreement_rate
0 google/gemini-2.0-flash-001|vision meta-llama/llama-3.2-3b-instruct:free|text 3120 1.0
1 google/gemini-2.0-flash-001|vision openai/gpt-4o-mini|text 3120 1.0
2 google/gemini-2.0-flash-001|vision openai/gpt-4o-mini|vision 3120 1.0
3 meta-llama/llama-3.2-3b-instruct:free|text openai/gpt-4o-mini|text 3120 1.0
4 meta-llama/llama-3.2-3b-instruct:free|text openai/gpt-4o-mini|vision 3120 1.0
5 openai/gpt-4o-mini|text openai/gpt-4o-mini|vision 3120 1.0

5. Peek at predictions

pred_path = OUT_DIR / "predictions.jsonl"
preds = [
    json.loads(line)
    for line in pred_path.read_text(encoding="utf-8").splitlines()
    if line.strip()
]
pred_df = pd.DataFrame(preds)
display(Markdown(f"**{len(pred_df)}** prediction rows"))
cols = [
    c
    for c in [
        "document_id",
        "label",
        "task",
        "modality",
        "model_id",
        "prediction",
        "error",
        "dry_run",
        "cache_hit",
    ]
    if c in pred_df.columns
]
display(pred_df[cols].head(12))

8320 prediction rows

document_id label task modality model_id prediction error dry_run cache_hit
0 test:imagesz/z/p/v/zpv93d00/508330972_50833097... letter classify vision openai/gpt-4o-mini letter None True False
1 test:imagesz/z/p/v/zpv93d00/508330972_50833097... letter extract_annotate vision openai/gpt-4o-mini {'document_type_guess': 'letter', 'title_or_su... None True False
2 test:imagesz/z/p/v/zpv93d00/508330972_50833097... letter classify vision google/gemini-2.0-flash-001 letter None True False
3 test:imagesz/z/p/v/zpv93d00/508330972_50833097... letter extract_annotate vision google/gemini-2.0-flash-001 {'document_type_guess': 'letter', 'title_or_su... None True False
4 test:imagesz/z/p/v/zpv93d00/508330972_50833097... letter classify text openai/gpt-4o-mini letter None True False
5 test:imagesz/z/p/v/zpv93d00/508330972_50833097... letter extract_annotate text openai/gpt-4o-mini {'document_type_guess': 'letter', 'title_or_su... None True False
6 test:imagesz/z/p/v/zpv93d00/508330972_50833097... letter classify text meta-llama/llama-3.2-3b-instruct:free letter None True False
7 test:imagesz/z/p/v/zpv93d00/508330972_50833097... letter extract_annotate text meta-llama/llama-3.2-3b-instruct:free {'document_type_guess': 'letter', 'title_or_su... None True False
8 train:imagese/e/u/v/euv68e00/2001213893.tif letter classify vision openai/gpt-4o-mini letter None True False
9 train:imagese/e/u/v/euv68e00/2001213893.tif letter extract_annotate vision openai/gpt-4o-mini {'document_type_guess': 'letter', 'title_or_su... None True False
10 train:imagese/e/u/v/euv68e00/2001213893.tif letter classify vision google/gemini-2.0-flash-001 letter None True False
11 train:imagese/e/u/v/euv68e00/2001213893.tif letter extract_annotate vision google/gemini-2.0-flash-001 {'document_type_guess': 'letter', 'title_or_su... None True False

Next steps

  • Download archive via section 0b / CLI (--confirm-writes-under-venv), then materialize
  • Set DRY_RUN=False for a live vision run
  • Lower MAX_PER_CLASS (e.g. 2) for a cheap smoke before the full 1040
  • Compare summary_classification.csv across model slugs; review extraction JSONL manually
  • Guide: RVL-CDIP SQL Index
Back to top