Home › Model › State

State¶

Per-model progress, tracked in a per-model state table. Opt in with state=State(...); bollhav records each unit of work's lifecycle and re-runs become resumable. A unit is one window for a batched temporal model, or the single one-shot row for a timeless model (or an unbatched temporal one) — so every temporality carries state, not just windowed models.

Status values¶

Each model's state lives in its own digest-named table in the one central bollhav schema — z_bollhav (prod) or z_bollhav_<suffix> (a suffixed run) — alongside the shared library and errors tables. The table name is a pure function of the model's canonical identity, so it never collides with the fixed library / errors names. The status column on it is one of:

Status	Meaning
`pending`	Queued to run. The user's loop iterates these.
`running`	Currently being processed. Set by `@state` immediately before invoking your `execute`. Visible in live dashboards.
`applied`	Completed successfully. Set by `@state` after a clean execute, or atomically by the staging flush.
`blocked`	Cannot run: an out-of-pipeline upstream isn't fulfilled. See `blocked_reason` for the BLOCK CODE.
`error`	Execute raised. Full details (type, message, traceback) are in the shared central `errors` table (`z_bollhav.errors`), keyed by `full_name`. Auto-retried on next run under `STATE_MODE=discover`.

Prefill and re-evaluation on rerun¶

Every run prefills first — it fills the state table with the contract. For every interval the contract declares (begin → end, or the latest complete tick for an open contract) it inserts a pending row that isn't already there, and leaves every existing row exactly as it is. This step is mode-independent and incremental: the table stays a complete mirror of the contract — as a new tick rolls over, its row simply appears — and since prior runs already laid down the rest, a run only inserts the new edge. Prefill never changes a row's status; it only makes sure the row exists. (Consequence: the first run on a brand-new model with a long contract materializes the whole declared range as pending, then runs only its window — the backlog is real and visible from run one.)

Execution is then window-scoped: a run does exactly the intervals in its resolved window (the latest complete tick, an explicit BACKFILL range, or — a no-dates backfill — the contract's declared range). Nothing outside the window leaks in. STATE_MODE decides only how the window's existing state is invalidated before it runs — orthogonal to prefill.

Under STATE_MODE=bulldozer (the default), the window's rows reset to pending and run — even already-applied ones — while everything outside the window is left untouched. "Do exactly this window, fresh." Boundaries are kept (chunk granularity unchanged). Note the split: prefill may have touched the whole contract, but the bulldozer reset is scoped to this run's window — redoing one day never disturbs the other applied days.

Under STATE_MODE=discover, nothing is invalidated: applied rows stay applied and the run takes only the window's still-outstanding rows — the idempotent, skip-what's-done mode. With no window it reconciles every outstanding row (the "no dates, let state find the work" catch-up):

pending rows whose upstreams have since regressed → blocked
blocked rows whose upstreams now satisfy → pending
error rows → pending (automatic retry)
running rows orphaned by a process crash → pending (automatic recovery)

Under STATE_MODE=torch, every state row for the model is deleted first; the prefill above then refills the whole contract range as pending. The wipe is always total — the window only scopes what runs now: with no window a torch runs the whole range (a clean full reload); with a BACKFILL window it runs just that slice now and leaves the rest pending for a later discover run to drain (nothing is orphaned — prefill refilled the whole contract). It's also how you change chunk granularity (hourly → monthly). Destructive (applied history is lost, so the next run reprocesses everything — safe for MERGE / recreate writes, but APPEND would duplicate); never fires during DRY_STATE; does not touch the model's data; scoped to the TAGS-matched models.

Concurrency: per-interval advisory locks¶

@state takes a per-interval Postgres advisory lock — keyed by a hash of the (model.full_name, since, until) triple — for every interval it processes. Two workers running on the same model but different intervals don't conflict; two workers racing on the same interval — only one wins, the other silently skips that interval and moves on.

This means you can scale horizontally on the same model:

# Terminal 1
TAGS=[orders] python main.py

# Terminal 2 — same model, splits the work
TAGS=[orders] python main.py

Both bootstraps see the same pending rows; both loops try the same intervals; the lock ensures each interval is processed exactly once. No special code in your loop — the decorator handles it.

Optional model-wide lock¶

The per-interval lock is the right default — two workers safely parallelize across different intervals of the same model. For stricter "one whole-pipeline run at a time per model" semantics (rare — usually only when interval ordering matters or your loop has cross-interval side effects), set State(allow_concurrent_runs=False):

from bollhav.model import Model, State

Model(state=State(allow_concurrent_runs=False), ...)

You don't take the lock yourself — @model_lifecycle does it for you. When allow_concurrent_runs=False, it takes a Postgres advisory lock keyed by the model's full_name at the start of the model's run and holds it (released in a finally) until the run ends, preventing any parallel processing on that model. If another run already holds it, the hook raises ModelLockedError — catch it around your run_model(...) call to skip:

from bollhav.model import ModelLockedError

for run in runs:
    try:
        run_model(run, conn)          # @model_lifecycle takes the lock
    except ModelLockedError:
        logger.warning("%s is locked; skipping", run.model.target.full_name)
        continue

With the default allow_concurrent_runs=True, no model-wide lock is taken and only the per-interval locks apply.

Errors¶

When an execute raises, @state does three things atomically:

Insert a row into the shared central errors table (z_bollhav.errors, or z_bollhav_<suffix>.errors) with full_name, run_id, error_type, error_message, traceback, created_at.
Flip the state row's status to error.
Re-raise the original exception so the caller sees it.

The errors table keeps full history across runs for every model — joinable with a model's state table on (since, until) for per-interval inspection or on run_id for per-invocation lookups. Because it's a single shared table, a global view is just SELECT … FROM z_bollhav.errors filtered (or grouped) by full_name — no UNION needed.

The one exception: if the staging flush already set state to applied (data is in target) and post-stage user code raises, we log the error but do not downgrade state to error. The write succeeded; the post-write code didn't.

Upstreams¶

A model's upstream is a list of contracts on other models, each checked before a unit of work runs. An unsatisfied contract → blocked. Contracts are only enforced here, in the state machine — so upstream requires state (declaring it without State(...) raises). See Upstream for the full picture.

Library and state colocation¶

The library lives in z_bollhav.library in the state DB. View and library=True models without their own state-DSN fall back to target.dsn_env_var — which is fine for single-DB setups (the common case where state and target share one Postgres database). If you split state to a separate database, library-only models also need their target DSN to point at that same instance, since they have no state.dsn_env_var to redirect them.

Disabling state entirely: `STATE_DISABLED`¶

Set STATE_DISABLED=true to force a pipeline to run with no state tracking, even when models declare state=State(...). Useful for:

Ad-hoc/dev runs against a fresh DB where the state tables don't exist
Quickly running the write path in isolation
Bypassing the state DB when you only want to test the read/write logic

When set: @load_models clears state and target.staging on every matched model, the state bootstrap and banner are skipped, @state becomes a passthrough, and write() uses the direct (non-staged) path. State tables aren't read from or written to during the run.

Recording an out-of-band load: `STATE_MARK_APPLIED`¶

STATE_MARK_APPLIED=true is the complement of STATE_DISABLED: it writes the state without running the data. It stamps the matched models' window intervals applied and exits — no read, no write, no MERGE, no target DDL.

The use case is exactly the pair: you loaded a table some other way (a STATE_DISABLED bulk load, a manual script, a one-off INSERT) and now want the state machine to know those intervals are done, so the daily incremental doesn't re-load them

Where the four combinations sit:

	state	data
normal run	✅ written	✅ run
`DRY_STATE`	✅ prefilled / classified	❌
`STATE_DISABLED`	❌	✅ run
`STATE_MARK_APPLIED`	✅ stamped applied	❌

Scope. It marks exactly the run's compute_intervals(run) — the [RUN_SINCE, RUN_UNTIL) (or LATEST) window, split by your chunk — and nothing else. Crucially it does not go through the normal "drain every actionable row" executor, so a leftover pending backlog from other intervals is left untouched. Match the chunk (INTERVAL_OVERRIDE) to how the data was actually loaded, since that's the grain of the rows it stamps

# you loaded 2025-11-06 out of band; now record it as done:
export TAGS="[FactCaseVariableDevice]"
export BACKFILL_ENABLED=true
export RUN_SINCE=2025-11-06T00:00:00
export RUN_UNTIL=2025-11-07T00:00:00
export INTERVAL_OVERRIDE=@daily        # the grain it was loaded at
export STATE_MARK_APPLIED=true
python src/main.py

It is an assertion, not a verification. bollhav does not check the data is actually there — it records that you say it is. If you're wrong, downstreams gate on a claim that isn't true. applied_at is set to now (the claim time, not when the data really landed), so a downstream Freshness check sees "just loaded." It logs a WARNING per model and shows a mark applied banner in the run summary, since it's a deliberate override.

Variable	Default	Effect
`STATE_MODE`	`bulldozer`	how much of the run's window to invalidate. `bulldozer` (default) resets the window's rows to `pending` and runs exactly them (boundaries kept); `discover` preserves `applied` and runs only the window's outstanding (no window → reconcile all); `torch` deletes every row then refills the contract — no window reloads the whole range, a window runs that slice now and defers the rest (the wipe is always total); for changing chunk granularity / a clean reload (destructive)
`STATE_DISABLED`	`false`	When `true`, force no-state behavior on every matched model (data without state)
`STATE_MARK_APPLIED`	`false`	When `true`, stamp the matched window's intervals `applied` without running them (state without data) — to record an out-of-band load. Scoped to `compute_intervals(run)`, never the backlog. An assertion, not a verification
`DRY_STATE`	`false`	When `true`, run the state bootstrap and print each model's resolved plan (would-run / applied / blocked), then exit without creating assets, writing data, or running model logic

Block codes¶

When bollhav inserts a state row as blocked instead of pending, the blocked_reason column always starts with a stable code like STATE_001:. Codes are namespaced by domain (STATE, future: WRITE, LIB, etc.), permanent once assigned, and never renumbered — so you can grep logs, build runbooks, and key alerts off them.

This page explains each code: what triggers it, what to do about it

STATE_001 — upstream not registered¶

Trigger. A staged, state-enabled model's upstream declares a model that has never been seen by @load_models. The library (z_bollhav.library) has no row for it, so bollhav can't reason about whether the upstream has produced data for the requested (since, until) window.

Example reason text.

STATE_001: upstream 'warehouse.orders' not registered

Why this is conservative. "Not registered" means nobody has ever run this model with @load_models against this state DB. It's distinct from "registered but not yet applied" (that's STATE_002) — a missing registration is a louder signal.

Remediation.

Run the upstream model in any pipeline that matches it. Once @load_models bootstraps it, the library row exists; the downstream's next bootstrap re-evaluates and flips the row from blocked to pending (under STATE_MODE=discover).
Double-check the spelling of upstream in the downstream's Model(...) definition — the string must match the upstream's Target.full_name exactly (<schema>.<name>).
If the upstream lives in a different state DB, see the note on cross-DB state in STATE.md. Today the library only resolves models in the same state DB.

STATE_002 — upstream has no applied row covering this interval¶

Trigger. The upstream IS in the library, but its state table has no row with status='applied' whose (since, until) either exactly matches or fully encapsulates the downstream's interval.