Provenance — inspect the generation/training audit log
Demo scale: uses a modest corpus (N=160) and --smoke training so it finishes on CPU in minutes while keeping eval metrics meaningful.
For production-scale runs, raise N and set SMOKE = False (DeBERTa-v3 / LayoutLMv3).
Not yet implemented (Phases 4–5): LoRA memo model training and the unified intake orchestrator. Stage B memos shown here are the synthetic targets those phases will train on.
0. Setup
Ensure the repo root is on sys.path, load config, and set demo knobs.
from __future__ import annotationsimport jsonimport loggingimport sysfrom collections import Counterfrom pathlib import Pathimport matplotlib.pyplot as pltimport pandas as pdimport yamlfrom IPython.display import Image, Markdown, display# Resolve repo root whether the kernel cwd is repo root or notebooks/CWD = Path.cwd().resolve()# Walk up from docs/notebooks/ (or notebooks/) until pyproject.toml is found.REPO_ROOT =next( (p for p in (CWD, *CWD.parents) if (p /"pyproject.toml").exists()),None,)assert REPO_ROOT isnotNone, f"Could not find repo root from {CWD}"ifstr(REPO_ROOT) notin sys.path: sys.path.insert(0, str(REPO_ROOT))from src.utils.config import Configfrom src.utils.io import load_jsonl, read_jsonlogging.basicConfig(level=logging.INFO, format="%(levelname)s%(name)s: %(message)s")cfg = Config.load()# --- Demo knobs ---N =48# portable docs demo size (matches data/notebook_demo/*_n48_seed42.jsonl)SEED =42SMOKE =True# DistilBERT smoke path; set False for DeBERTa / LayoutLMv3DEMO = REPO_ROOT /"data"/"notebook_demo"DEMO.mkdir(parents=True, exist_ok=True)print(f"repo: {REPO_ROOT}")print(f"python: {sys.version.split()[0]}")print(f"LLM key set: {bool(cfg.openrouter_api_key)}")print(f"N={N} SEED={SEED} SMOKE={SMOKE}")print(f"demo outputs → {DEMO}")
Classification targets come from taxonomy/acord_form_categories.yaml (ACORD-inspired; includes a few non-ACORD claim attachments). adjuster_memo is a Stage B output type and is excluded from Stage A classifier training.
tax = yaml.safe_load(cfg.taxonomy_path.read_text(encoding="utf-8"))tax_df = pd.DataFrame( [ {"label": c["label"],"lifecycle_stage": c["lifecycle_stage"],"acord_forms": ", ".join(c.get("acord_forms") or []) or"—","description": c["description"].strip().split("\n")[0][:90] +"…", }for c in tax["categories"] ])display(tax_df)print(f"{len(tax_df)} categories | classifier excludes: adjuster_memo")
label
lifecycle_stage
acord_forms
description
0
application_commercial
underwriting_intake
125, 126, 140, 127, 130, 131, 146
Commercial insurance applications and coverage...
1
application_personal
underwriting_intake
90, 80, 83, 84
Personal lines applications: auto, homeowners,...
2
certificate_evidence
policy_servicing
24, 25, 27, 28
Certificates and evidence of insurance -- proo...
3
loss_notice
claim_intake
1, 2
First notice of loss documents for property an...
4
claims_correspondence
claim_processing
—
Free-text correspondence related to an open cl...
5
adjuster_memo
claim_processing
—
Internal case memos written by adjusters summa...
6
policy_change_endorsement
policy_servicing
101
Requests to modify an active policy: endorseme...
7
repair_estimate
claim_processing
—
Third-party repair/replacement cost estimates ...
8
supporting_evidence
claim_processing
—
Police reports, photos (described, not the ima...
9 categories | classifier excludes: adjuster_memo
2. Characteristic profiles
Bundled priors under data/profiles/ seed generation. run_profiler refreshes them from any ingested Hub samples in data/raw/ (optional; skipped by default).
from src.generation.characteristic_profiler import run_profilerprofile_versions = run_profiler(cfg)print("Profile versions:")for name, ver in profile_versions.items():print(f" {name}: {ver}")dist = read_json(cfg.profiles_dir /"insurance_distributions.json")legal = read_json(cfg.profiles_dir /"legal_style_profile.json")ocr = read_json(cfg.profiles_dir /"ocr_noise_profile.json")fig, axes = plt.subplots(1, 2, figsize=(11, 4))doc_w = dist["document_type_weights"]axes[0].barh(list(doc_w.keys()), list(doc_w.values()), color="#2c5f7c")axes[0].set_title("Document-type prior weights")axes[0].set_xlabel("weight")loss_w = dist["loss_type_weights"]axes[1].barh(list(loss_w.keys()), list(loss_w.values()), color="#6b8f71")axes[1].set_title("Loss-type prior weights")axes[1].set_xlabel("weight")plt.tight_layout()plt.show()print("Legal style (vocab only — never used as class labels):")print(" ", ", ".join(legal.get("vocabulary_ngrams", [])[:8]), "…")print("OCR noise rates:", {k: ocr[k] for k in ("char_substitution_rate", "char_deletion_rate", "char_insertion_rate")})
Legal style (vocab only — never used as class labels):
proximate cause, material fact, coverage determination, reservation of rights, duty to defend, indemnify and hold harmless, preponderance of the evidence, reasonable care …
OCR noise rates: {'char_substitution_rate': 0.025, 'char_deletion_rate': 0.008, 'char_insertion_rate': 0.006}
3. Skeleton sampling
Each skeleton is a schema-validated claim intermediate (data/schemas/claim_skeleton.schema.json) with policy, loss, parties, and financials. Fixed train/val/test splits are written to data/synthetic/splits.json.
from src.generation.skeleton_sampler import run_samplersk_path = run_sampler(cfg, n=N, seed=SEED, out=DEMO /f"skeletons_n{N}_seed{SEED}.jsonl")skeletons = load_jsonl(sk_path)splits = read_json(cfg.splits_path)print(f"Wrote {len(skeletons)} skeletons → {sk_path}")print(f"Splits: train={len(splits['train'])} val={len(splits['val'])} test={len(splits['test'])}")type_counts = Counter(s["document_type"] for s in skeletons)display(pd.DataFrame(type_counts.most_common(), columns=["document_type", "count"]))sample = skeletons[0]display(Markdown(f"### Sample skeleton — `{sample['claim_id']}` / `{sample['document_type']}`"))print(json.dumps(sample, indent=2)[:1800])
Wrote 48 skeletons → /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/data/notebook_demo/skeletons_n48_seed42.jsonl
Splits: train=34 val=7 test=7
Renders insurance document text from each skeleton using layout + surface + legal-style profiles. If OPENROUTER_API_KEY is set, an LLM is tried first with template fallback.
from src.generation.stage_a_document_gen import run_stage_adoc_path = DEMO /f"documents_n{N}_seed{SEED}.jsonl"ifnot doc_path.exists(): doc_path = run_stage_a(cfg, sk_path, out=doc_path, seed=SEED)else:print(f"reusing existing {doc_path}")docs = load_jsonl(doc_path)modes = Counter(d["generation_mode"] for d in docs)print(f"Documents: {len(docs)} modes={dict(modes)}")print(f"→ {doc_path}")doc = docs[0]display(Markdown(f"### Sample document — `{doc['document_type']}` (`{doc['generation_mode']}`)"))print(doc["text"][:2200])
Dear Claimant
Claim Number: CLM-2025-666564
ACORD Form: N/A
RE:
Adjuster Name: Taylor Brown
Claim Status: under review (standard)
CLAIM STATUS
Adjuster Name: Taylor Brown
Claim Status: under review (standard)
REQUESTED ACTION
Adjuster Name: Taylor Brown
Claim Status: under review (standard)
NEXT STEPS
Policy Number: AZ-974628-L
Named Insured: Harper Young
NARRATIVE
Subject to further investigation Reference is made to proximate cause. The loss is described as: smoke odor infiltration from neighboring unit fire. Complexity assessment: standard.
Claims Department
Prepared for claim CLM-2025-666564
5. Stage B — adjuster memo generation
Produces memo training targets conditioned on the skeleton + Stage A text (issue/rule/application/conclusion style from the legal profile).
ADJUSTER MEMO — CLM-2025-666564
To: Claims File
From: Taylor Brown
Re: Harper Young / AZ-974628-L
Summary
Subject to further investigation the reported other loss on 2025-10-27 under commercial_general_liability in AZ.
Facts
- Location: 2764 Hillcrest Ln, AZ
- Description: smoke odor infiltration from neighboring unit fire
- Police report: yes
- Injuries: no
- Estimated damage: $13,918.95
- Deductible: $5,000.00
- Current reserve: $14,405.66
Analysis
The record supports a finding that the claim file for CLM-2025-666564 is standard, because smoke odor infiltration from neighboring unit fire. Alternative explanations are less consistent with the Stage A source document and reported loss details.
Next Steps
- Confirm coverage grant/denial points in writing
- Update reserve if investigation changes exposure
- Request any missing supporting evidence
Source document type: claims_correspondence
6. OCR noise injection
Applies character/word-level OCR artifacts from ocr_noise_profile.json and builds a multi-document bundle index for grouped claim files.
from src.generation.noise_injection import run_noisenoisy_path = DEMO /f"noisy_n{N}_seed{SEED}.jsonl"ifnot noisy_path.exists(): noisy_path = run_noise(cfg, doc_path, out=noisy_path, seed=SEED)else:print(f"reusing existing {noisy_path}")noisy_docs = load_jsonl(noisy_path)# Side-by-side clean vs noisy excerptclean_excerpt = docs[0]["text"][:500]noisy_excerpt = noisy_docs[0]["text"][:500]display(Markdown("### Clean vs OCR-noisy (first ~500 chars)"))fig, axes = plt.subplots(1, 2, figsize=(14, 5))for ax, title, text in ( (axes[0], "Clean", clean_excerpt), (axes[1], "Noisy", noisy_excerpt),): ax.axis("off") ax.set_title(title) ax.text(0.01, 0.99, text, va="top", ha="left", family="monospace", fontsize=8, wrap=True, transform=ax.transAxes)plt.tight_layout()plt.show()index_path = cfg.noisy_output_dir /f"multi_doc_index_from_{doc_path.stem}.json"if index_path.exists(): index = read_json(index_path)print(f"Multi-doc groups: {len(index)}")if index: gid, members =next(iter(index.items()))print(f" example {gid}: {members}")
Multi-doc groups: 4
example GRP-67154: ['CLM-2025-440870::loss_notice::13', 'CLM-2025-440870::supporting_evidence::14', 'CLM-2025-440870::claims_correspondence::15']
7. Classification — prepare dataset
Maps Stage A documents → taxonomy labels using the fixed splits. Writes train/val/test.jsonl + label2id.json.
from src.classification.prepare_dataset import prepare as prepare_classificationclf_prepared = prepare_classification(doc_path, cfg, out_dir=DEMO /"classification_prepared")summary = read_json(clf_prepared /"summary.json")label2id = read_json(clf_prepared /"label2id.json")print("Prepared:", clf_prepared)print("Split sizes:", {k: summary[k] for k in ("train", "val", "test")})print("Labels:", summary["labels"])train_rows = load_jsonl(clf_prepared /"train.jsonl")display(pd.Series([r["label"] for r in train_rows]).value_counts().rename("train_count").to_frame())
# Extraction failure modes
Observed / expected hard fields on synthetic forms:
- **Dates (`date_of_loss`, `effective_date`)**: OCR digit confusions (0/O, 1/l) and format variation.
- **Dollar amounts (`estimated_damage`, `deductible`, `reserve_set`)**: commas, `$` glyphs, and OCR substitutions.
- **Free-text location / narrative-adjacent values**: long spans bleed into neighboring fields under BIO labeling.
- **Noisy variants**: token F1 and field exact-match drop vs clean renders; partial match remains more stable.
## Measured hard-field partial match
- `date_of_loss`: 0.000
- `estimated_damage`: 0.000
- `deductible`: 0.000
- `reserve_set`: 0.000
- `location`: 0.000
## Noisy stress summary
- token_macro_f1: 0.063
- field_exact_mean: 0.000
- field_partial_mean: 0.000
14. End-to-end inference on one document
Classify a held-out document, run token-level field extraction, and show the paired Stage B memo (the Phase 4 summarization target — not yet a trained local generator).
import torchfrom transformers import ( AutoModelForSequenceClassification, AutoModelForTokenClassification, AutoTokenizer,)test_rows = load_jsonl(clf_prepared /"test.jsonl")assert test_rows, "Need at least one test row — raise N if empty"row = test_rows[0]record_id = row["record_id"]# --- Classify ---clf_tok = AutoTokenizer.from_pretrained(str(clf_model_dir))clf_model = AutoModelForSequenceClassification.from_pretrained(str(clf_model_dir))clf_model.eval()with torch.no_grad(): inputs = clf_tok(row["text"], return_tensors="pt", truncation=True, max_length=512) pred_id =int(torch.argmax(clf_model(**inputs).logits, dim=-1).item())pred_label = clf_model.config.id2label[pred_id]# --- Extract fields from matching rendered page ---ext_test = {r["record_id"]: r for r in load_jsonl(ext_prepared /"test.jsonl")}ext_row = ext_test.get(record_id) or load_jsonl(ext_prepared /"train.jsonl")[0]ext_tok = AutoTokenizer.from_pretrained(str(ext_model_dir))ext_model = AutoModelForTokenClassification.from_pretrained(str(ext_model_dir))ext_model.eval()id2label = {int(v): k for k, v in read_json(ext_model_dir /"label2id.json").items()}enc = ext_tok( ext_row["tokens"], is_split_into_words=True, return_tensors="pt", truncation=True, max_length=256,)with torch.no_grad(): pred_ids = torch.argmax(ext_model(**enc).logits, dim=-1)[0].tolist()word_ids = enc.word_ids(batch_index=0)aligned = ["O"] *len(ext_row["tokens"])seen =set()for idx, wid inenumerate(word_ids):if wid isNoneor wid in seen:continue seen.add(wid) aligned[wid] = id2label.get(int(pred_ids[idx]), "O")# Decode BIO spansfrom src.extraction.evalimport _decode_entitiespred_ents = _decode_entities(ext_row["tokens"], aligned)gold_ents = _decode_entities( ext_row["tokens"], [id2label.get(int(i), "O") for i in ext_row["labels"]])# --- Paired memo ---memo_by_id = {m["record_id"]: m for m in memos}paired_memo = memo_by_id.get(record_id)display(Markdown(f"### Record `{record_id}`"))print(f"Gold label: {row['label']}")print(f"Predicted label: {pred_label}")print()print("Extracted fields (pred vs gold):")fields =sorted(set(pred_ents) |set(gold_ents))for f in fields: pvals = pred_ents.get(f) or [] gvals = gold_ents.get(f) or [] pred_s = (pvals[0][:60] if pvals else"—") gold_s = (gvals[0][:60] if gvals else"—")print(f" {f:22s} pred={pred_s}")print(f" {'':22s} gold={gold_s}")if paired_memo: display(Markdown("### Paired Stage B memo (summarization training target)"))print(paired_memo["memo_text"][:1800])
Paired Stage B memo (summarization training target)
ADJUSTER MEMO — CLM-2023-720893
To: Claims File
From: Jordan Nguyen
Re: Skyler Martinez / MD-765046-I
Summary
The claimant alleges the reported wind_hail loss on 2023-12-06 under commercial_property in MD.
Facts
- Location: 8380 Sunset Ave, MD
- Description: wind-driven tree limb through living room window
- Police report: no
- Injuries: no
- Estimated damage: $6,258.27
- Deductible: $500.00
- Current reserve: $6,753.30
Analysis
The record supports a finding that the claim file for CLM-2023-720893 is fraud_flagged, because wind-driven tree limb through living room window. Alternative explanations are less consistent with the Stage A source document and reported loss details.
Next Steps
- Confirm coverage grant/denial points in writing
- Update reserve if investigation changes exposure
- Request any missing supporting evidence
Source document type: application_commercial
15. Provenance audit log
Every generation and training stage appends a record to data/provenance_log.jsonl.
prov_path = cfg.provenance_log_pathif prov_path.exists(): prov = load_jsonl(prov_path)# Show the most recent entries from this notebook run recent = prov[-12:] display(pd.DataFrame( [ {"stage": r.get("stage"),"source": str(r.get("source", ""))[:50],"model": r.get("model"),"prompt_version": r.get("prompt_version"), }for r in recent ] ))print(f"Total provenance records: {len(prov)} → {prov_path}")else:print("No provenance log yet.")
stage
source
model
prompt_version
0
docie_pipeline
/Users/morningstar/Desktop/Cold_Storage/smol-d...
document_processing→document_classification→in...
docie_fig1_v1
1
docie_pipeline
/Users/morningstar/Desktop/Cold_Storage/smol-d...
document_processing→document_classification→in...
docie_fig1_v1
2
docie_pipeline
/Users/morningstar/Desktop/Cold_Storage/smol-d...
document_processing→document_classification→in...
docie_fig1_v1
3
docie_pipeline
/Users/morningstar/Desktop/Cold_Storage/smol-d...
document_processing→document_classification→in...
docie_fig1_v1
4
rvl_cdip_build
aharley/rvl_cdip
NaN
rvl_cdip_sql_v1
5
classification_train_random_forest
/Users/morningstar/Desktop/Cold_Storage/smol-d...
sklearn.RandomForestClassifier
random_forest_multilayer_v1
6
characteristic_profiling
data/profiles
NaN
profiles_v1
7
skeleton_sampling
insurance_distributions.json
NaN
skeleton_sampler_v1
8
characteristic_profiling
data/profiles
NaN
profiles_v1
9
skeleton_sampling
insurance_distributions.json
NaN
skeleton_sampler_v1
10
classification_train
/Users/morningstar/Desktop/Cold_Storage/smol-d...
distilbert-base-uncased
classification_v1
11
extraction_train
/Users/morningstar/Desktop/Cold_Storage/smol-d...
distilbert-base-uncased
extraction_v1
Total provenance records: 89 → /Users/morningstar/Desktop/Cold_Storage/smol-doc-analyzer/data/provenance_log.jsonl
