Sample Document Corpus Walkthrough — src/storage/

End-to-end showcase of the synthetic medical-bill + salvage-claim corpus added for analysis and fine-tuning without proprietary American Family Insurance data.

This notebook covers the new pipelines:

  1. Rich skeleton schemas (medical + salvage)
  2. Realistic sample generation (HCFA / UB-04 / LOG / sales / towing)
  3. SQLite store seeding
  4. Claim bundles (multi-document files)
  5. Export → DICIE inference
  6. Import existing fixtures / eval gold
Section What you will see
0 Setup
1 Schemas & taxonomies
2 Generate synthetic corpus
3 Seed SQLite store
4 Inspect claims, docs, fields
5 Claim bundles
6 Export for DICIE / training
7 Run DICIE on corpus export
8 Import fixtures into the store
9 Provenance

Canonical docs: docs/sample_document_corpus.md, docs/data_provenance.md.

All documents are fictional. Carrier branding (American Family / AmFam) is used only to simulate intake surfaces — no proprietary claim files are loaded.

0. Setup

from __future__ import annotations

import json
import logging
import sqlite3
import sys
import time
from collections import Counter
from pathlib import Path

import matplotlib.pyplot as plt
import pandas as pd
from IPython.display import Markdown, display

CWD = Path.cwd().resolve()
# Walk up from docs/notebooks/ (or notebooks/) until pyproject.toml is found.
REPO_ROOT = next(
    (p for p in (CWD, *CWD.parents) if (p / "pyproject.toml").exists()),
    None,
)
assert REPO_ROOT is not None, f"Could not find repo root from {CWD}"
if str(REPO_ROOT) not in sys.path:
    sys.path.insert(0, str(REPO_ROOT))

from src.docie import DociePipeline
from src.docie.applications import list_applications, load_application
from src.docie.eval import evaluate_application
from src.docie.pipeline import run_file
from src.storage import DocumentStore
from src.storage.sample_generator import generate_claim_bundle, generate_corpus
from src.storage.schema import DDL, SCHEMA_VERSION
from src.storage.training import (
    fit_tfidf_random_forest,
    prepare_both_applications,
)
from src.storage.types import ClaimRecord, DocumentRecord, FieldRecord
from src.utils.config import Config
from src.utils.io import load_jsonl, read_json, write_json

logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s")
cfg = Config.load()

DEMO = REPO_ROOT / "data" / "notebook_demo" / "sample_corpus"
DEMO.mkdir(parents=True, exist_ok=True)
DB_PATH = DEMO / "documents.db"
EXPORTS = DEMO / "exports"
EXPORTS.mkdir(parents=True, exist_ok=True)
PREPARED = DEMO / "prepared"
MODELS = DEMO / "models"
MODELS.mkdir(parents=True, exist_ok=True)

SEED = 42
print(f"repo:     {REPO_ROOT}")
print(f"demo db:  {DB_PATH}")
print(f"exports:  {EXPORTS}")
print(f"schema v: {SCHEMA_VERSION}")
print(f"apps:     {list_applications()}")
repo:     /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer
demo db:  /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/data/notebook_demo/sample_corpus/documents.db
exports:  /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/data/notebook_demo/sample_corpus/exports
schema v: 1
apps:     ['acord', 'medical_bills', 'salvage_claims']

1. Schemas & taxonomies

med_schema = read_json(REPO_ROOT / "data" / "schemas" / "medical_bill_skeleton.schema.json")
sal_schema = read_json(REPO_ROOT / "data" / "schemas" / "salvage_document_skeleton.schema.json")
print("Medical schema title:", med_schema["title"])
print("  required:", med_schema["required"])
print("Salvage schema title:", sal_schema["title"])
print("  required:", sal_schema["required"])

for app in ("medical_bills", "salvage_claims"):
    profile = load_application(app)
    print(f"\n[{app}] labels={profile.labels}")
    print(f"  extraction_fields={profile.extraction_fields}")
Medical schema title: MedicalBillSkeleton
  required: ['claim_id', 'document_type', 'carrier', 'patient', 'provider', 'services', 'financials']
Salvage schema title: SalvageDocumentSkeleton
  required: ['claim_id', 'document_type', 'carrier', 'vehicle', 'parties', 'financials']

[medical_bills] labels=['hcfa', 'ub04', 'other']
  extraction_fields=['claim_id', 'name', 'dob', 'patient_id', 'address']

[salvage_claims] labels=['log', 'sales', 'other']
  extraction_fields=['claim_id', 'vin', 'year', 'make', 'model']

2. Generate synthetic corpus

corpus = generate_corpus(
    seed=SEED,
    medical_per_type=6,
    salvage_per_type=6,
    bundles_per_app=2,
    include_canonical_fixtures=True,
)
print(f"claims={len(corpus.claims)}  documents={len(corpus.documents)}")

type_counts = Counter((d.application, d.document_type) for d in corpus.documents)
pd.DataFrame(
    [{"application": a, "document_type": t, "count": n} for (a, t), n in sorted(type_counts.items())]
)
claims=46  documents=54
application document_type count
0 medical_bills hcfa 9
1 medical_bills other 9
2 medical_bills ub04 9
3 salvage_claims log 9
4 salvage_claims other 9
5 salvage_claims sales 9

Peek at a Letter of Guarantee and an HCFA

log_doc = next(d for d in corpus.documents if d.document_type == "log")
hcfa_doc = next(d for d in corpus.documents if d.document_type == "hcfa")
display(Markdown(f"**{log_doc.document_id}** (`{log_doc.document_type}`)"))
print(log_doc.text[:800])
print("---")
display(Markdown(f"**{hcfa_doc.document_id}** (`{hcfa_doc.document_type}`)"))
print(hcfa_doc.text[:800])
print("\nLOG ground truth:", log_doc.ground_truth_fields())
print("HCFA ground truth:", hcfa_doc.ground_truth_fields())

sal-log-001 (log)

LETTER OF GUARANTEE
First National Bank Lienholder Services
This letter guarantees that the insurer reimbursement will pay the bank first.
Claim Number: CLM-2024-100200
VIN: 1HGCM82633A004352
Year: 2018
Make: Honda
Model: Accord
Payoff Amount: $4,250.00
---

med-hcfa-001 (hcfa)

HCFA CMS-1500 HEALTH INSURANCE CLAIM FORM
Physician or Supplier Information
Patient Name: Jane Q Public
Date of Birth: 03/14/1988
Patient ID: PID-778812
Claim Number: CLM-2024-551122
Address: 100 Oak Avenue, Madison WI 53703
Carrier Name: American Family
Diagnosis and procedure codes follow.

LOG ground truth: {'claim_id': 'CLM-2024-100200', 'vin': '1HGCM82633A004352', 'year': '2018', 'make': 'Honda', 'model': 'Accord'}
HCFA ground truth: {'claim_id': 'CLM-2024-551122', 'name': 'Jane Q Public', 'dob': '03/14/1988', 'patient_id': 'PID-778812', 'address': '100 Oak Avenue, Madison WI 53703'}

3. Seed SQLite store

if DB_PATH.exists():
    DB_PATH.unlink()
store = DocumentStore(DB_PATH)
n = store.bulk_upsert(corpus.documents, claims=corpus.claims)
store.add_provenance(
    stage="notebook_seed",
    source="sample_document_corpus_walkthrough",
    detail={"documents": n, "seed": SEED},
)
summary = store.summary()
display(summary)
pd.DataFrame(summary["by_application_type"])
{'db_path': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/data/notebook_demo/sample_corpus/documents.db',
 'schema_version': 1,
 'claims': 46,
 'documents': 54,
 'fields': 270,
 'by_application_type': [{'application': 'medical_bills',
   'document_type': 'hcfa',
   'count': 9},
  {'application': 'medical_bills', 'document_type': 'other', 'count': 9},
  {'application': 'medical_bills', 'document_type': 'ub04', 'count': 9},
  {'application': 'salvage_claims', 'document_type': 'log', 'count': 9},
  {'application': 'salvage_claims', 'document_type': 'other', 'count': 9},
  {'application': 'salvage_claims', 'document_type': 'sales', 'count': 9}]}
application document_type count
0 medical_bills hcfa 9
1 medical_bills other 9
2 medical_bills ub04 9
3 salvage_claims log 9
4 salvage_claims other 9
5 salvage_claims sales 9

4. Inspect claims, documents, and fields

docs = store.list_documents(limit=8)
pd.DataFrame(
    [
        {
            "document_id": d.document_id,
            "application": d.application,
            "document_type": d.document_type,
            "claim_id": d.claim_id,
            "split": d.split,
            "source_kind": d.source_kind,
            "n_fields": len(d.fields),
        }
        for d in docs
    ]
)
document_id application document_type claim_id split source_kind n_fields
0 med-hcfa-001 medical_bills hcfa CLM-2024-551122 test canonical_fixture 5
1 med-ub04-002 medical_bills ub04 CLM-2024-660033 test canonical_fixture 5
2 med-other-003 medical_bills other CLM-SYN-MED-OTHER-003 test canonical_fixture 5
3 sal-log-001 salvage_claims log CLM-2024-100200 test canonical_fixture 5
4 sal-sales-002 salvage_claims sales CLM-2024-100201 test canonical_fixture 5
5 sal-other-003 salvage_claims other CLM-2024-100202 test canonical_fixture 5
6 med-hcfa-100 medical_bills hcfa CLM-2022-719176 test synthetic_seed 5
7 med-ub04-100 medical_bills ub04 CLM-2024-946335 test synthetic_seed 5 
canonical = store.get_document("sal-log-001")
assert canonical is not None
claim = store.get_claim(canonical.claim_id) if canonical.claim_id else None
print("carrier:", claim.carrier_name if claim else None)
print("fields:", canonical.ground_truth_fields())
print(canonical.text)
carrier: American Family
fields: {'claim_id': 'CLM-2024-100200', 'vin': '1HGCM82633A004352', 'year': '2018', 'make': 'Honda', 'model': 'Accord'}
LETTER OF GUARANTEE
First National Bank Lienholder Services
This letter guarantees that the insurer reimbursement will pay the bank first.
Claim Number: CLM-2024-100200
VIN: 1HGCM82633A004352
Year: 2018
Make: Honda
Model: Accord
Payoff Amount: $4,250.00

5. Claim bundles (multi-document salvage / medical files)

import random

rng = random.Random(7)
claim, bundle_docs = generate_claim_bundle(rng, application="salvage_claims", bundle_index=99)
store.upsert_claim(claim)
for d in bundle_docs:
    store.upsert_document(d)

bundled = store.list_documents(claim_id=claim.claim_id)
print(f"claim {claim.claim_id} has {len(bundled)} documents:")
for d in bundled:
    print(f"  - {d.document_id:28s} {d.document_type:8s}  title={d.title}")
claim CLM-2024-258176 has 3 documents:
  - sal-bundle99-log             log       title=Letter of Guarantee
  - sal-bundle99-sales           sales     title=Salvage Sales Receipt
  - sal-bundle99-other           other     title=Towing / Storage Invoice

6. Export for DICIE / training

salvage_docie = EXPORTS / "salvage_docie.jsonl"
medical_docie = EXPORTS / "medical_docie.jsonl"
clf_all = EXPORTS / "classification_all.jsonl"
ext_all = EXPORTS / "extraction_all.jsonl"

n_sal = store.export_jsonl(salvage_docie, format="docie", application="salvage_claims")
n_med = store.export_jsonl(medical_docie, format="docie", application="medical_bills")
n_clf = store.export_jsonl(clf_all, format="classification")
n_ext = store.export_jsonl(ext_all, format="extraction")
print({"salvage_docie": n_sal, "medical_docie": n_med, "classification": n_clf, "extraction": n_ext})

sample = load_jsonl(salvage_docie)[0]
meta = {k: sample[k] for k in sample if k != "text"}
print(json.dumps(meta, indent=2)[:800])
print("text preview:\n", sample["text"][:400])
{'salvage_docie': 30, 'medical_docie': 27, 'classification': 57, 'extraction': 57}
{
  "record_id": "sal-log-001",
  "application": "salvage_claims",
  "document_type": "log",
  "ground_truth_fields": {
    "claim_id": "CLM-2024-100200",
    "vin": "1HGCM82633A004352",
    "year": "2018",
    "make": "Honda",
    "model": "Accord"
  },
  "claim_id": "CLM-2024-100200",
  "split": "test",
  "metadata": {
    "carrier_style": "american_family_simulation"
  }
}
text preview:
 LETTER OF GUARANTEE
First National Bank Lienholder Services
This letter guarantees that the insurer reimbursement will pay the bank first.
Claim Number: CLM-2024-100200
VIN: 1HGCM82633A004352
Year: 2018
Make: Honda
Model: Accord
Payoff Amount: $4,250.00

7. Run DICIE on corpus export

out_pred = DEMO / "docie_salvage_predictions.jsonl"
run_file(
    salvage_docie,
    out_pred,
    application="salvage_claims",
    cfg=cfg,
    run_ocr=False,
    limit=20,
)
summary_path = out_pred.with_suffix(".summary.json")
print("summary →", summary_path)
pred_summary = read_json(summary_path)
display(pred_summary)

preds = load_jsonl(out_pred)
pd.DataFrame(
    [
        {
            "record_id": p["record_id"],
            "document_type": p.get("document_type"),
            "needs_human_review": p.get("needs_human_review"),
            "fields": p.get("fields"),
        }
        for p in preds[:12]
    ]
)
summary → /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/data/notebook_demo/sample_corpus/docie_salvage_predictions.summary.json
{'n': 20,
 'application': 'salvage_claims',
 'chain': ['document_processing',
  'document_classification',
  'information_extraction',
  'output_aggregation'],
 'needs_human_review': 1,
 'label_counts': {'log': 7, 'sales': 7, 'other': 6},
 'avg_classification_confidence': 0.9012500000000001,
 'avg_extraction_confidence': 0.8299999999999998}
record_id document_type needs_human_review fields
0 sal-log-001 log False {'claim_id': 'CLM-2024-100200', 'vin': '1HGCM8...
1 sal-sales-002 sales False {'claim_id': 'CLM-2024-100201', 'vin': '1FADP3...
2 sal-other-003 other True {'claim_id': 'CLM-2024-100202', 'vin': None, '...
3 sal-log-100 log False {'claim_id': 'CLM-2024-497519', 'vin': '5NPE24...
4 sal-sales-100 sales False {'claim_id': 'CLM-2025-801978', 'vin': '1HGCM8...
5 sal-other-100 other False {'claim_id': 'CLM-2026-778998', 'vin': '3VWDP7...
6 sal-log-101 log False {'claim_id': 'CLM-2022-618317', 'vin': '3VWDP7...
7 sal-sales-101 sales False {'claim_id': 'CLM-2026-486945', 'vin': 'KM8J3C...
8 sal-other-101 other False {'claim_id': 'CLM-2024-288158', 'vin': 'KM8J3C...
9 sal-log-102 log False {'claim_id': 'CLM-2023-950544', 'vin': '2T1BUR...
10 sal-sales-102 sales False {'claim_id': 'CLM-2025-822858', 'vin': '2T1BUR...
11 sal-other-102 other False {'claim_id': 'CLM-2026-666211', 'vin': 'WBA8E9...

Single-document DICIE on a generated LOG

pipe = DociePipeline(application="salvage_claims", cfg=cfg, run_ocr=False)
demo_log = store.get_document("sal-log-001")
prediction = pipe.process(record_id=demo_log.document_id, text=demo_log.text)
display(prediction.response_payload())
{'record_id': 'sal-log-001',
 'application': 'salvage_claims',
 'document_type': 'log',
 'classification_confidence': 0.91,
 'fields': {'claim_id': 'CLM-2024-100200',
  'vin': '1HGCM82633A004352',
  'year': '2018',
  'make': 'Honda',
  'model': 'Accord'},
 'extraction_confidence': 0.85,
 'needs_human_review': False,
 'flags': ['extract_heuristic']}

8. Import existing fixtures / eval gold into the store

fixtures = REPO_ROOT / "tests" / "fixtures" / "sample_docie_documents.jsonl"
eval_set = REPO_ROOT / "data" / "eval" / "docie_eval_set.jsonl"
n_fix = store.import_docie_jsonl(fixtures, source_kind="test_fixture")
n_eval = store.import_docie_jsonl(eval_set, source_kind="docie_eval_gold") if eval_set.exists() else 0
print(f"imported fixtures={n_fix} eval_gold={n_eval}")
store.summary()
imported fixtures=6 eval_gold=24
{'db_path': '/Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/data/notebook_demo/sample_corpus/documents.db',
 'schema_version': 1,
 'claims': 56,
 'documents': 75,
 'fields': 375,
 'by_application_type': [{'application': 'medical_bills',
   'document_type': 'hcfa',
   'count': 12},
  {'application': 'medical_bills', 'document_type': 'other', 'count': 12},
  {'application': 'medical_bills', 'document_type': 'ub04', 'count': 12},
  {'application': 'salvage_claims', 'document_type': 'log', 'count': 13},
  {'application': 'salvage_claims', 'document_type': 'other', 'count': 13},
  {'application': 'salvage_claims', 'document_type': 'sales', 'count': 13}]}

9. Provenance

with sqlite3.connect(DB_PATH) as conn:
    conn.row_factory = sqlite3.Row
    rows = conn.execute(
        "SELECT event_id, stage, source, detail_json, created_at "
        "FROM provenance_events ORDER BY event_id"
    ).fetchall()
pd.DataFrame([dict(r) for r in rows])
event_id stage source detail_json created_at
0 1 notebook_seed sample_document_corpus_walkthrough {"documents": 54, "seed": 42} 1.784791e+09
1 2 import_docie_jsonl /Users/morningstar/Desktop/Cold_Storage/smol-d... {"imported": 6, "source_kind": "test_fixture"} 1.784791e+09
2 3 import_docie_jsonl /Users/morningstar/Desktop/Cold_Storage/smol-d... {"imported": 24, "source_kind": "docie_eval_go... 1.784791e+09

Next notebooks

Back to top