DataSynth 5.30 → 5.33.2: Eval-Driven Realism, 570× Memory Ceiling Lift, Framework-Aware Classification

Five DataSynth releases (5.30, 5.31, 5.32, 5.33, plus two 5.33.x patches) shipped between the 5.29 SOTA structural-fidelity round and today. The VynFi API now pins 5.33.2 as the final engine version. The theme across the line: stop adding realism features, start proving the ones we have.

5.30 — Tier-A + Tier-B SOTA round (eval-driven)

Two parallel tiers. Tier-A wires shipped infrastructure (P3 graph motifs, P4 velocity rules, Z-tail mass) into the Sajja exact eval — the research-grade detection-rate benchmark we use as our headline fraud-detection score. Tier-B adds the engine-side levers the eval surfaced as gaps:

• **B1 — per-source IET sampler refinement.** Inter-event-time sampling now respects per-source code paths with overdispersion control. The corpus shows distinct IET distributions per business process (a P2P workflow has different timing dynamics than a journal-entry adjustment cycle); the engine now matches.
• **B2 — ConsolidationOutlierPass (config flag, default 0.001).** Reshapes a Bernoulli(rate) fraction of JEs into multi-100-line bridge-account postings (10 seeded suspense/clearing accounts) — lifts the synthetic `relational_score` p99/max toward the corpus's heavy tail without distorting the median. Two-phase implementation: Phase 1a is the schema field + injector, Phase 1b is the multi-line bridge-account expansion. CoA constant fix in Phase 1b ensures the suspense accounts are realistic for the chart of accounts in scope.
• **B3 — per-process fraud rate distributions.** Different business processes (P2P, O2C, payroll, treasury) have radically different baseline fraud rates. The 5.30 lever lets you tune per-process rates as opt-in overrides for the global fraud_rate baseline. Closes #153.
• **A3 — Z-tail mass tuned 0.30 → 0.15.** Reduces tail-only divergence in the BF composite score; surfaced by multi-seed validation.

The throughline: every change in 5.30 either added measurement (Sajja exact-eval rules) or a lever (B1/B2/B3) the eval can now act on. The methodology is closed-loop — measure, propose, regenerate, re-measure.

5.31 — C1 streaming aggregate + C2 multi-period

C1 — the memory unblock

Group consolidation at 2 000-entity scale was hitting an OOM ceiling: 218 GB peak. The 5.31 C1 round is seven phases of incremental working-set collapse, with two load-bearing fixes:

• **Phase 6 — `JeNetworkEdgeBuilder` + streaming JSON-array Visitor.** Replaces `serde_json::from_slice::<Vec<JournalEntry>>` per entity. Per-iteration allocator fragmentation drops from ~178 MB / entity to ~75 KB / entity. This is the line item that did most of the work.
• **Phase 7 — `elimination_amount` defensive fallback + IC-pair-id gate on `ReversedAmountStrategy::can_apply`.** Defends against partial-IC corruption from anomaly strategies in future shards.

Practical impact for VynFi: the group-audit workflow that has been waiting for this is unblocked. The current PROD blocker (#59 — bundling the new `datasynth-data` runtime binary alongside the compile-time crate bump that landed this week) is now genuinely the only thing standing between Wave 3's IAS 36 / IAS 29 / multi-period features and prod-enablement.

C2 — multi-period closing-to-opening

Three pieces (`cd4236d0`, `47dbdfff`, `e32fe019`) that thread per-entity closing TBs into the next-period shard run:

• `project_closing_to_opening` converter — carries BS accounts, zeroes P&L, absorbs net income into Retained Earnings (one-sided normalised for loss years).
• `datasynth_group::shard::multi_period::build_opening_balances_from_prior` + `run_shard_chained` — wires the converter into shard execution.
• CLI surface: `datasynth-data group shard --prior-period-shards` + `--prior-period-framework` (the framework knob lets per-entity classification follow the entity's local GAAP).

The same code path also fixes a silent bug in the existing `generate_standalone_chain` runner — it was using a BS-only `extract_opening_balances` that dropped P&L on Adjusted TBs without absorbing it. Five-year chains were silently losing prior-year net income. Now fixed.

VynFi's multi-period chain feature (the Wave 1 surface from DS 5.5) gets the framework knob automatically via the bundled binary bump. No portal changes required.

5.32 — C3 adversarial calibration loop

Six pieces across the `datasynth_eval::calibration` module and the `datasynth-data calibrate` CLI. The closed-loop framework: drive the engine's tunable knobs so a chosen gap metric (versus a reference corpus) converges, with multi-seed variance averaging and explicit safety rails:

• **Calibration objective + knob types.** `CalibrationObjective` minimises one of three BF headline scalars (`composite_bf_score`, `composite_bf_median`, `composite_bf_volume_corrected`); aggregates across seeds into `(mean, std)` so the loop can reject sub-noise-floor improvements (per the v5.31 T3 multi-seed methodology finding).
• **Iteration controller.** `CalibrationLoop::step` runs one multi-seed generate → eval → propose → accept/reject cycle. Three rollback policies (Revert / Keep / HalveDamping) handle steps that worsen loss.
• **Stock proposers.** Greedy coordinate-descent with per-knob direction memory; both ± directions tried before a knob is marked exhausted. Round-robin baseline for sanity.
• **History persistence.** `CalibrationHistory` snapshots the trajectory + best-tracker after every step; atomic JSON write; schema-versioned load + resume hook.
• **Safety rails.** `OscillationDetector` flags sign-alternating Δ across a configurable window; `KnobClipDiagnostics` surfaces frequently-clipped knobs; `WallClockBudget` wraps an Instant-based budget for overnight runs.
• **`datasynth-data calibrate` CLI.** Subcommand with `--config`, `--reference`, `--objective`, `--max-iter`, `--seeds`, `--target`, `--resume`, `--dry-run`. Six default knobs spanning the most impactful v5.30 SOTA tunables.

The orchestrator-backed `Evaluator` that runs full generation per `(knobs, seed)` is a VM-validated follow-up (Piece 4b). Without `--dry-run`, the current CLI emits a clear 'Piece 4b not yet wired' exit; the dry-run smoke-tests argparse + setup without orchestration. This is an internal research feature; nothing for VynFi to surface in the API yet.

5.33 / 5.33.1 / 5.33.2 — framework-aware classification + chain-mode persistence

A three-step fix triggered by the v5.32 3-year medium-chain semantics check. The check surfaced three pre-existing defects on German SKR + French PCG entities:

• **Defect A** — every `TrialBalanceLine.account_type` was hard-coded to `Asset` in the per-entity TB writer.
• **Defect B** — `category_from_account_code` shipped a US-only prefix table that mis-routed SKR (`0xxx` Fixed Assets, `4xxx` Revenue, `8xxx` tax) and PCG (`5x` Cash, `6x` Expenses, `7x` Revenue).
• **Defect C** — `is_balanced` compared YTD-BS against period-only-P&L — a structurally meaningless test that always returned false for any post-month-1 TB.

Combined, these drove a 55% ACME_EU debit-credit gap and a ~32% consolidated A vs L+E+NCI divergence on the 3-year German entity run. Three patches close the gap:

5.33 — per-entity TB writer framework-aware

`PeriodTrialBalance` now carries a `framework: String` field populated by the orchestrator's new `resolve_framework_str` helper (country-first, framework-label fallback). `into_canonical` consumes it and dispatches to `FrameworkAccounts::classify_account_type` (per-line `account_type`) + `AccountCategory::from_account_code_with_framework` (per-line `category`). Aliases accepted: `us_gaap` / `ifrs` / `dual_reporting` / `french_gaap` / `pcg` / `german_gaap` / `hgb`.

5.33.1 — consolidated-FS aggregator framework-aware

The per-entity fix in 5.33 closed Defects A/B/C at the per-entity layer, but VM validation showed the consolidated FS gap stayed at ~32% — same shape, different code path. The aggregator was still using a US-only numeric-range table. `AggregatedAccount` now carries an `account_type: AccountType` field (with `#[serde(default)]` for backward-compat) populated by `accumulate_tb`. `classify_bs_section` is replaced by `classify_bs_section_from_account` which reads the top-level Asset / Liability / Equity decision from `account.account_type` rather than guessing from the code prefix.

5.33.2 — `opening_balances.json` persistence in chain mode

A separate silent bug surfaced by the same FINDINGS run: `balance/opening_balances.json` was missing across all three years of the 3-year medium chain. Phase 3b in the orchestrator was being silently skipped because `cfg.balance.generate_opening_balances` defaulted to `false` and the per-entity chain config builder never force-enabled it. With the flag off the v5.3 `ShardContext.opening_balances` carry-forward also no-op'd at the early return — Year N+1 chains were dropping the prior-year closings the chain helper had gone to the trouble of computing.

5.33.2 stamps `cfg.balance.generate_opening_balances = true` for every shard and restructures `phase_opening_balances` so the carryover branch runs unconditionally. VM validation on the same 3-year chain: all 9 entity-year `opening_balances.json` files present; 99.5%+ of Y_N+1 opening balances match Y_N closes by magnitude (the expected convention difference: TB `closing_balance` is debit-normal, `opening_balances.json::balances` is natural-side); 1 expected mismatch per entity per year on account `3200` (retained earnings — Y_N+1 opening RE = Y_N opening RE + Y_N net income, while the Y_N closing TB shows pre-closing-entry RE; that's correct closing-of-books behaviour).

Surface scan — what VynFi had to change

The compile-time DS pin bump (PR #7, merged 2026-05-28) was the cleanest bump we've shipped — zero diagnostics, zero call-site fixes. The 5.30-5.33.2 surface changes all live in the runtime layer (per-entity TB writer, consolidated FS aggregator, calibration module) that the VynFi API doesn't call directly. The translation layer sits one floor up.

One lockfile fix was needed for `aes 0.9.0` (transitive via `zip 8.6.0` ← `datasynth-generators/fingerprint`) — it got yanked from crates.io shortly after DS 5.33.2 published, so `cargo audit --deny warnings` failed in CI. A `cargo update -p aes@0.9.0 --precise 0.9.1` resolved it.

What's next

Two follow-up surfaces for VynFi: a wizard toggle for ConsolidationOutlierPass, and a wizard knob for per-process fraud rate overrides — both open to every account, credits are the only meter. Both mirror the DS 5.29 `foreign_currency_rate` toggle pattern we shipped a few weeks ago.

The single remaining prod-enablement gate is bundling the `datasynth-data` 5.33.2 runtime binary alongside the compile-time crate bump that just landed — release-pipeline owner's call. Once that ships, the IAS 36 / IAS 29 / multi-period Wave 3 surfaces go live for every account, and the C1 streaming-aggregate memory ceiling unlock makes 2 000-entity group consolidations actually runnable in prod.

5.33.2 is the engine version VynFi is pinning long-term. From here, the work moves from feature-adoption to surfacing the levers in the wizard and proving the realism claims with the Sajja eval methodology landing alongside.