# Telemetry

Structured telemetry on polars and hive-partitioned parquet: logs, run and artifact indexes, and sweep progress.

## femforge.telemetry

Structured telemetry on polars and hive-partitioned parquet: logs, run and artifact indexes, sweep progress.

Modules:

- **`artifacts`** –

  The artifact index: store sidecars read into parquet by directory mtime, with a fluent query over them.
- **`cli`** –

  The telemetry half of the `femforge` command: build the indexes, list runs and sweeps, query the logs, compact the store.
- **`context`** –

  The sweep point a thread or task is working on, carried on a contextvar and bound onto every log record it emits.
- **`convergence`** –

  One run's convergence log read into a frame, with the sif-authored targets that give its progress a meaning.
- **`hive`** –

  Append-only hive-partitioned parquet: one shard per writer per partition, never rewritten under a reader.
- **`journal`** –

  The write-ahead half of the store: one JSONL journal per process, flushed per record and checkpointed into parquet.
- **`paths`** –

  Where telemetry lives: the root resolution rule and the typed hive stores under it.
- **`runs`** –

  The run index: bundles read into parquet by manifest mtime, a fluent query over them, and the live overlay from the run locks.
- **`schemas`** –

  The polars schemas of every telemetry table, with column names single-sourced as StrEnums.
- **`sink`** –

  The loguru sink that lands every record in parquet, plus the query surface over what it wrote.
- **`sweep`** –

  Sweep progress: one row per launch, one journalled event per point transition, and the query that reads them back.
- **`tail`** –

  Incremental reads of a growing text file, so a viewer re-reads only the bytes it has not seen.
- **`tap`** –

  An in-process live view of the log stream: a bounded ring buffer plus asyncio fan-out to the workbench.

Classes:

- **`ArtifactIndex`** –

  The parquet index of one local artifact store: an incremental `update` over its committed directories, plus `query`.
- **`ArtifactQuery`** –

  A fluent filter over the artifact index; each method returns a new query and nothing runs until `frame`.
- **`SweepPoint`** –

  Which sweep, and which point of it, the current context belongs to.
- **`ConvergenceTargetsFile`** –

  The iteration caps and tolerances one run's sif authored, written beside its convergence log so the log reads in context.
- **`LoopTarget`** –

  One solver loop's authored iteration cap and tolerance, in the form the bundle persists them.
- **`HiveStore`** –

  One hive-partitioned parquet table: a typed `scan`, an atomic `append` of one shard per partition, and a settling `compact`.
- **`Journal`** –

  A process's own append-only JSONL journal: every record is written and flushed immediately, then folded into parquet by `checkpoint`.
- **`Telemetry`** –

  A telemetry root and the typed hive store for each table under it.
- **`TelemetryDir`** –

  The subdirectories of a telemetry root: one per table, plus the journal directory the live tables are tailed from.
- **`TelemetryRootMissing`** –

  Neither an explicit path nor `FEMFORGE_TELEMETRY` names a telemetry root.
- **`RunIndex`** –

  The parquet index of one runs directory: an incremental `update` over its bundles, the `record` a driver appends per run, and `query`.
- **`RunQuery`** –

  A fluent filter over the run index; each method returns a new query and nothing runs until `frame`.
- **`ArtifactColumn`** –

  The artifact index's columns: one row per stored artifact, keyed by `key`.
- **`ConvergenceColumn`** –

  The keys `elmer.runtime.solve` writes into `convergence.jsonl`, one sample per solver iteration.
- **`ConvergenceLoop`** –

  Which Elmer iteration loop a convergence sample belongs to: nonlinear or steady state.
- **`LiveRunColumn`** –

  The columns of the in-flight run table built from the run lock owners; never persisted.
- **`LogColumn`** –

  The log table's columns; the sink emits these names and every query selects by them.
- **`RunColumn`** –

  The run index's columns: one row per run bundle, keyed by `run_id`.
- **`RunSource`** –

  How a run index row was produced: a filesystem scan of the bundles, or the Driver reporting live.
- **`SweepColumn`** –

  The sweep table's columns: one row per sweep launch.
- **`SweepEventColumn`** –

  The sweep event table's columns: one row per point state transition.
- **`SweepState`** –

  The lifecycle states a sweep point moves through; each transition is one sweep event row.
- **`LogQuery`** –

  A fluent filter over the log frame; each method returns a new query and nothing runs until `page`.
- **`ParquetLogSink`** –

  Loguru sink writing every record to a JSONL journal at once and to hive-partitioned parquet in batches.
- **`SweepJournal`** –

  One sweep's writer: the launch and closing rows in the sweeps table, every point transition through this process's own journal.
- **`SweepQuery`** –

  One sweep read back: its launch row, every point transition (settled shards plus live journals), and the progress summary.
- **`SweepSummary`** –

  Where a sweep stands: how many points sit in each state, how fast they are landing, and how long the rest should take.
- **`TextTail`** –

  A byte-offset cursor over one growing text file; a truncation rewinds it and an unterminated last line is held back until it ends.
- **`LiveLogTap`** –

  Loguru sink keeping the last records in memory and pushing each new one onto every asyncio subscriber queue.

Functions:

- **`current_sweep_point`** –

  The sweep point in scope, or None outside any `sweep_point` block.
- **`sweep_point`** –

  Mark this context as one sweep point, binding both ids onto every log record emitted inside it.
- **`scan_convergence`** –

  A run's `convergence.jsonl` as a typed frame, carrying each loop's first relative change and how far it has come toward its tolerance.
- **`coerce`** –

  Cast a loosely-typed frame (JSON text, a partial row set) onto `schema`, parsing date/time strings and filling missing columns with nulls.
- **`journal_glob`** –

  The glob matching every process's journal for the table `name`.
- **`recover_journals`** –

  Checkpoint and delete the journals of processes that are gone, so a crash costs no records; returns the journals recovered.
- **`scan_journals`** –

  A frame over every matching journal's records, coerced onto `schema`; lines that are not JSON records are skipped.
- **`require_root`** –

  The telemetry root a reader needs, or `TelemetryRootMissing` naming both ways to provide one.
- **`resolve_root`** –

  The telemetry root: `explicit` if given, else `FEMFORGE_TELEMETRY`, else None because telemetry is off.
- **`live_runs`** –

  The runs claimed right now under `runs_dir`, read from each `<run_id>.lock/owner.json` with the age of its heartbeat.
- **`scan_logs`** –

  Every log record: the settled parquet shards unioned with what is still only in the live journals.
- **`sweep_runs`** –

  Every sweep the root knows, newest first and one row each (a finished sweep's closing row wins).

Attributes:

- **`TELEMETRY_ENV`** –

  The env var naming the telemetry root; unset and unpassed means telemetry is off and polars is never imported.
- **`ARTIFACT_INDEX_SCHEMA`** –

  One row per `artifact.json` sidecar; the tool-version columns mirror `ToolVersions`.
- **`CONVERGENCE_SCHEMA`** –

  Read straight from a bundle's `convergence.jsonl`; per-run and never centralised, so it has no partitions.
- **`LIVE_RUN_SCHEMA`** –

  Built per call from the `owner.json` of each `runs/<run_id>.lock`, so it is always current rather than stored.
- **`LOG_SCHEMA`** –

  One row per log record; `extra` is the JSON of everything bound that has no column of its own.
- **`RUN_INDEX_SCHEMA`** –

  One row per run bundle; the index re-reads a bundle only when `manifest_mtime` moves, and dedupes on `run_id`.
- **`SWEEP_EVENT_SCHEMA`** –

  Append-only point transitions; the latest state per `point_key` is the sweep's live progress.
- **`SWEEP_RUN_SCHEMA`** –

  One row per sweep; a row is rewritten (and deduped on `sweep_id`) when the sweep finishes.

### TELEMETRY\_ENV

```python
TELEMETRY_ENV = 'FEMFORGE_TELEMETRY'
```

The env var naming the telemetry root; unset and unpassed means telemetry is off and polars is never imported.

### ARTIFACT\_INDEX\_SCHEMA

```python
ARTIFACT_INDEX_SCHEMA = pl.Schema({ArtifactColumn.KIND: pl.String, ArtifactColumn.KEY: pl.String, ArtifactColumn.NAME: pl.String, ArtifactColumn.STORE_ROOT: pl.String, ArtifactColumn.CREATED_AT: TIMESTAMP, ArtifactColumn.SPEC_TYPE: pl.String, ArtifactColumn.PRODUCER_VERSION: pl.Int32, ArtifactColumn.PRODUCER_HOST: pl.String, ArtifactColumn.PRODUCER_PID: pl.Int32, ArtifactColumn.FILE_COUNT: pl.Int32, ArtifactColumn.TOTAL_BYTES: pl.Int64, ArtifactColumn.FILES: pl.List(pl.String), ArtifactColumn.SPEC: pl.String, ArtifactColumn.FEMFORGE: pl.String, ArtifactColumn.CODE_SHA: pl.String, ArtifactColumn.GMSH: pl.String, ArtifactColumn.ELMER: pl.String, ArtifactColumn.DIR_MTIME: TIMESTAMP, ArtifactColumn.INDEXED_AT: TIMESTAMP})
```

One row per `artifact.json` sidecar; the tool-version columns mirror `ToolVersions`.

### CONVERGENCE\_SCHEMA

```python
CONVERGENCE_SCHEMA = pl.Schema({ConvergenceColumn.WALL_TIME: pl.Float64, ConvergenceColumn.SOLVER: pl.String, ConvergenceColumn.LOOP: pl.Enum(categories(ConvergenceLoop)), ConvergenceColumn.ITER: pl.Int32, ConvergenceColumn.NRM: pl.Float64, ConvergenceColumn.RELC: pl.Float64})
```

Read straight from a bundle's `convergence.jsonl`; per-run and never centralised, so it has no partitions.

### LIVE\_RUN\_SCHEMA

```python
LIVE_RUN_SCHEMA = pl.Schema({LiveRunColumn.RUN_ID: pl.String, LiveRunColumn.HOST: pl.String, LiveRunColumn.PID: pl.Int32, LiveRunColumn.STARTED_AT: TIMESTAMP, LiveRunColumn.HEARTBEAT_AT: TIMESTAMP, LiveRunColumn.HEARTBEAT_AGE_S: pl.Float64, LiveRunColumn.STALE: pl.Boolean})
```

Built per call from the `owner.json` of each `runs/<run_id>.lock`, so it is always current rather than stored.

### LOG\_SCHEMA

```python
LOG_SCHEMA = pl.Schema({LogColumn.DATE: pl.Date, LogColumn.PROCESS: pl.String, LogColumn.TIME: TIMESTAMP, LogColumn.LEVEL: pl.Enum(categories(LogLevel)), LogColumn.LEVEL_NO: pl.Int16, LogColumn.NAME: pl.String, LogColumn.FUNCTION: pl.String, LogColumn.LINE: pl.Int32, LogColumn.MESSAGE: pl.String, LogColumn.RUN_ID: pl.String, LogColumn.SWEEP_ID: pl.String, LogColumn.POINT_KEY: pl.String, LogColumn.HOST: pl.String, LogColumn.PID: pl.Int32, LogColumn.THREAD: pl.String, LogColumn.EXCEPTION: pl.String, LogColumn.EXTRA: pl.String})
```

One row per log record; `extra` is the JSON of everything bound that has no column of its own.

### RUN\_INDEX\_SCHEMA

```python
RUN_INDEX_SCHEMA = pl.Schema({RunColumn.MONTH: pl.Date, RunColumn.RUN_ID: pl.String, RunColumn.CACHE_KEY: pl.String, RunColumn.RUNS_DIR: pl.String, RunColumn.STATUS: pl.Enum(literal_categories(ManifestStatus)), RunColumn.CREATED_AT: TIMESTAMP, RunColumn.FINISHED_AT: TIMESTAMP, RunColumn.CURRENT_STAGE: pl.String, RunColumn.STAGES_DONE: pl.Int32, RunColumn.GIT_SHA: pl.String, RunColumn.ELMER_VERSION: pl.String, RunColumn.PHYSICS: pl.String, RunColumn.ELEMENT_COUNT: pl.Int64, RunColumn.MIN_ELEMENT_SIZE: pl.Float64, RunColumn.MAX_ELEMENT_SIZE: pl.Float64, RunColumn.DESCRIPTION: pl.String, RunColumn.TAGS: pl.String, RunColumn.NOTES: pl.String, RunColumn.MESH_NAME: pl.String, RunColumn.MESH_KEY: pl.String, RunColumn.PUBLISHED: pl.Boolean, RunColumn.HAS_RESULT: pl.Boolean, RunColumn.HAS_VTU: pl.Boolean, RunColumn.CLEANUPS: pl.Int32, RunColumn.BUNDLE_BYTES: pl.Int64, RunColumn.SWEEP_ID: pl.String, RunColumn.POINT_KEY: pl.String, RunColumn.MANIFEST_MTIME: TIMESTAMP, RunColumn.INDEXED_AT: TIMESTAMP, RunColumn.SOURCE: pl.Enum(categories(RunSource)), RunColumn.MODEL_DIM: pl.Int8, RunColumn.ARTIFACT_KEYS: pl.List(pl.String), RunColumn.FORCE_Y: pl.Float64, RunColumn.TEMPERATURE_MAX: pl.Float64, RunColumn.WALL_TIME: pl.Float64})
```

One row per run bundle; the index re-reads a bundle only when `manifest_mtime` moves, and dedupes on `run_id`.

### SWEEP\_EVENT\_SCHEMA

```python
SWEEP_EVENT_SCHEMA = pl.Schema({SweepEventColumn.SWEEP_ID: pl.String, SweepEventColumn.POINT_KEY: pl.String, SweepEventColumn.POINT_NAME: pl.String, SweepEventColumn.STATE: pl.Enum(categories(SweepState)), SweepEventColumn.AT: TIMESTAMP, SweepEventColumn.HOST: pl.String, SweepEventColumn.PID: pl.Int32, SweepEventColumn.ORIGIN: pl.String, SweepEventColumn.ERROR: pl.String, SweepEventColumn.DURATION_S: pl.Float64})
```

Append-only point transitions; the latest state per `point_key` is the sweep's live progress.

### SWEEP\_RUN\_SCHEMA

```python
SWEEP_RUN_SCHEMA = pl.Schema({SweepColumn.MONTH: pl.Date, SweepColumn.SWEEP_ID: pl.String, SweepColumn.DESCRIPTION: pl.String, SweepColumn.STARTED_AT: TIMESTAMP, SweepColumn.FINISHED_AT: TIMESTAMP, SweepColumn.TOTAL: pl.Int32, SweepColumn.SKIPPED: pl.Int32, SweepColumn.MAX_WORKERS: pl.Int32, SweepColumn.HOST: pl.String, SweepColumn.PID: pl.Int32, SweepColumn.PROCESS: pl.String, SweepColumn.GIT_SHA: pl.String, SweepColumn.INDEXED_AT: TIMESTAMP})
```

One row per sweep; a row is rewritten (and deduped on `sweep_id`) when the sweep finishes.

### ArtifactIndex

```python
ArtifactIndex(telemetry: Telemetry, store: LocalArtifactStore)
```

The parquet index of one local artifact store: an incremental `update` over its committed directories, plus `query`.

Methods:

- **`update`** –

  Index every artifact directory whose mtime moved since the last pass — all of them when `rebuild` — and return the rows appended.
- **`query`** –

  A fluent query over the index, reading no sidecar until `records` asks for one.

#### update

```python
update(*, rebuild: bool = False) -> int
```

Index every artifact directory whose mtime moved since the last pass — all of them when `rebuild` — and return the rows appended.

#### query

```python
query() -> ArtifactQuery
```

A fluent query over the index, reading no sidecar until `records` asks for one.

### ArtifactQuery

```python
ArtifactQuery(lazy: LazyFrame, store: LocalArtifactStore, limit: int | None = None)
```

A fluent filter over the artifact index; each method returns a new query and nothing runs until `frame`.

Methods:

- **`where`** –

  A new query narrowed by `predicate`; the other filters are named shorthands for this.
- **`kind`** –

  Artifacts of any of `kinds`.
- **`spec_type`** –

  Artifacts whose spec model's `module:QualName` ends with `name`, so the bare class name is enough.
- **`since`** –

  Artifacts published at or after `when`.
- **`key_prefix`** –

  Artifacts whose key starts with `prefix`.
- **`newest`** –

  Keep only the `count` most recently published artifacts, applied after the duplicate rows collapse.
- **`frame`** –

  The matching rows, newest first, one per stored artifact (the latest row wins).
- **`records`** –

  The matching artifacts read back from the store, skipping any whose directory is gone.

Attributes:

- **`lazy`** (`LazyFrame`) –

  The artifact index rows the query filters.
- **`store`** (`LocalArtifactStore`) –

  The store `records` reads the matching artifacts back from.
- **`limit`** (`int | None`) –

  How many of the newest artifacts to keep; None keeps them all.

#### lazy

```python
lazy: LazyFrame
```

The artifact index rows the query filters.

#### store

```python
store: LocalArtifactStore
```

The store `records` reads the matching artifacts back from.

#### limit

```python
limit: int | None = None
```

How many of the newest artifacts to keep; None keeps them all.

#### where

```python
where(predicate: Expr) -> ArtifactQuery
```

A new query narrowed by `predicate`; the other filters are named shorthands for this.

#### kind

```python
kind(*kinds: str) -> ArtifactQuery
```

Artifacts of any of `kinds`.

#### spec\_type

```python
spec_type(name: str) -> ArtifactQuery
```

Artifacts whose spec model's `module:QualName` ends with `name`, so the bare class name is enough.

#### since

```python
since(when: datetime) -> ArtifactQuery
```

Artifacts published at or after `when`.

#### key\_prefix

```python
key_prefix(prefix: str) -> ArtifactQuery
```

Artifacts whose key starts with `prefix`.

#### newest

```python
newest(count: int) -> ArtifactQuery
```

Keep only the `count` most recently published artifacts, applied after the duplicate rows collapse.

#### frame

```python
frame() -> DataFrame
```

The matching rows, newest first, one per stored artifact (the latest row wins).

#### records

```python
records() -> list[StoredArtifact]
```

The matching artifacts read back from the store, skipping any whose directory is gone.

### SweepPoint

Bases: `NamedTuple`

Which sweep, and which point of it, the current context belongs to.

Attributes:

- **`sweep_id`** (`str`) –

  The sweep's id.
- **`point_key`** (`str`) –

  The point's key within the sweep.

#### sweep\_id

```python
sweep_id: str
```

The sweep's id.

#### point\_key

```python
point_key: str
```

The point's key within the sweep.

### ConvergenceTargetsFile

Bases: `BaseModel`

The iteration caps and tolerances one run's sif authored, written beside its convergence log so the log reads in context.

Config:

- `use_attribute_docstrings`: `True`

Fields:

- `loops` (`list[LoopTarget]`)

#### loops

```python
loops: list[LoopTarget]
```

One entry per (solver, loop) the solved stack authored targets for.

#### of

```python
of(targets: ConvergenceTargets) -> Self
```

The persistable form of the mapping `convergence_targets` builds from a solver stack.

#### load

```python
load(path: str | Path) -> Self
```

Load a bundle's `convergence_targets.json` (pass the bundle directory or the file itself); a bundle without one reads as no targets.

#### write

```python
write(sim_dir: str | Path) -> Path
```

Write the targets into `sim_dir` as `convergence_targets.json` and return the path.

#### targets

```python
targets() -> ConvergenceTargets
```

The renderer's mapping again, keyed the way Elmer tags ComputeChange lines.

### LoopTarget

Bases: `BaseModel`

One solver loop's authored iteration cap and tolerance, in the form the bundle persists them.

Config:

- `use_attribute_docstrings`: `True`

Fields:

- `solver` (`str`)
- `loop` (`ConvergenceLoop`)
- `max_iterations` (`int | None`)
- `tolerance` (`float | None`)

#### solver

```python
solver: str
```

The sif `Equation` name Elmer tags this loop's ComputeChange lines with, normalized.

#### loop

```python
loop: ConvergenceLoop
```

Which loop the targets apply to: the nonlinear inner loop or the steady-state outer one.

#### max\_iterations

```python
max_iterations: int | None = None
```

The loop's iteration cap; None when the sif left Elmer's default.

#### tolerance

```python
tolerance: float | None = None
```

`Δ_tol` `[-]`: The relative-change tolerance the loop converges to; None when the sif left Elmer's default.

### HiveStore

```python
HiveStore(root: Path, schema: Schema, partition_keys: Sequence[str])
```

One hive-partitioned parquet table: a typed `scan`, an atomic `append` of one shard per partition, and a settling `compact`.

Parameters:

- **`root`** (`Path`) –

  The table's directory, holding one directory level per partition key.
- **`schema`** (`Schema`) –

  Every column of the table, partition columns included.
- **`partition_keys`** (`Sequence[str]`) –

  The columns whose values name the partition directories, outermost first.

Methods:

- **`shards`** –

  Every shard file under the root, in sorted order.
- **`partitions`** –

  The directories holding shards, one per distinct partition-key combination.
- **`scan`** –

  A lazy frame over every shard, each typed onto the schema, or a typed empty frame when nothing has been written.
- **`append`** –

  Write `frame` as one new shard per partition and return the shard paths; existing shards are untouched.
- **`compact`** –

  Fold each partition's settled shards into one merged shard and delete the originals; returns the merged shards.

Attributes:

- **`hive_schema`** (`Schema`) –

  The partition columns, whose values live in the directory names rather than the file bodies.
- **`body_schema`** (`Schema`) –

  The columns actually stored inside each shard: everything but the partition keys.

#### hive\_schema

```python
hive_schema: Schema
```

The partition columns, whose values live in the directory names rather than the file bodies.

#### body\_schema

```python
body_schema: Schema
```

The columns actually stored inside each shard: everything but the partition keys.

#### shards

```python
shards() -> list[Path]
```

Every shard file under the root, in sorted order.

#### partitions

```python
partitions() -> list[Path]
```

The directories holding shards, one per distinct partition-key combination.

#### scan

```python
scan() -> LazyFrame
```

A lazy frame over every shard, each typed onto the schema, or a typed empty frame when nothing has been written.

#### append

```python
append(frame: DataFrame) -> list[Path]
```

Write `frame` as one new shard per partition and return the shard paths; existing shards are untouched.

#### compact

```python
compact(settle_s: float = SETTLE_S) -> list[Path]
```

Fold each partition's settled shards into one merged shard and delete the originals; returns the merged shards.

### Journal

```python
Journal(live_dir: Path, name: str)
```

A process's own append-only JSONL journal: every record is written and flushed immediately, then folded into parquet by `checkpoint`.

Methods:

- **`write`** –

  Append one record and flush it, so a reader tailing the journal sees it before the next parquet flush.
- **`checkpoint`** –

  Append the journal's records to `store` and truncate it; returns the shards written.
- **`close`** –

  Close the journal handle and delete the (now empty) file; idempotent.

#### write

```python
write(row: Mapping[str, object]) -> None
```

Append one record and flush it, so a reader tailing the journal sees it before the next parquet flush.

#### checkpoint

```python
checkpoint(store: HiveStore) -> list[Path]
```

Append the journal's records to `store` and truncate it; returns the shards written.

#### close

```python
close() -> None
```

Close the journal handle and delete the (now empty) file; idempotent.

### Telemetry

```python
Telemetry(root: str | Path)
```

A telemetry root and the typed hive store for each table under it.

Methods:

- **`stores`** –

  Every table under this root, so a maintenance pass reaches them all without naming them one by one.

Attributes:

- **`live`** (`Path`) –

  The directory holding each process's write-ahead JSONL journal.
- **`logs`** (`HiveStore`) –

  Every log record, partitioned by date and process.
- **`runs`** (`HiveStore`) –

  The run index, partitioned by the month a run was created in.
- **`artifacts`** (`HiveStore`) –

  The artifact index, partitioned by artifact kind.
- **`sweeps`** (`HiveStore`) –

  A launch row per sweep plus a closing row once it finishes, partitioned by the month it started in.
- **`sweep_events`** (`HiveStore`) –

  Per-point state transitions, partitioned by sweep so one sweep's progress reads one directory.

#### live

```python
live: Path
```

The directory holding each process's write-ahead JSONL journal.

#### logs

```python
logs: HiveStore
```

Every log record, partitioned by date and process.

#### runs

```python
runs: HiveStore
```

The run index, partitioned by the month a run was created in.

#### artifacts

```python
artifacts: HiveStore
```

The artifact index, partitioned by artifact kind.

#### sweeps

```python
sweeps: HiveStore
```

A launch row per sweep plus a closing row once it finishes, partitioned by the month it started in.

#### sweep\_events

```python
sweep_events: HiveStore
```

Per-point state transitions, partitioned by sweep so one sweep's progress reads one directory.

#### stores

```python
stores() -> list[HiveStore]
```

Every table under this root, so a maintenance pass reaches them all without naming them one by one.

### TelemetryDir

Bases: `StrEnum`

The subdirectories of a telemetry root: one per table, plus the journal directory the live tables are tailed from.

Attributes:

- **`LOGS`** –

  The log table.
- **`RUNS`** –

  The run index.
- **`ARTIFACTS`** –

  The artifact index.
- **`SWEEPS`** –

  The sweep table: a launch row per sweep, plus a closing row once it finishes.
- **`SWEEP_EVENTS`** –

  The sweep event table: one row per point transition.
- **`LIVE`** –

  The write-ahead journals the live tables are tailed from.

#### LOGS

```python
LOGS = 'logs'
```

The log table.

#### RUNS

```python
RUNS = 'runs'
```

The run index.

#### ARTIFACTS

```python
ARTIFACTS = 'artifacts'
```

The artifact index.

#### SWEEPS

```python
SWEEPS = 'sweeps'
```

The sweep table: a launch row per sweep, plus a closing row once it finishes.

#### SWEEP\_EVENTS

```python
SWEEP_EVENTS = 'sweep_events'
```

The sweep event table: one row per point transition.

#### LIVE

```python
LIVE = 'live'
```

The write-ahead journals the live tables are tailed from.

### TelemetryRootMissing

Bases: `ValueError`

Neither an explicit path nor `FEMFORGE_TELEMETRY` names a telemetry root.

### RunIndex

```python
RunIndex(telemetry: Telemetry, runs_dir: str | Path)
```

The parquet index of one runs directory: an incremental `update` over its bundles, the `record` a driver appends per run, and `query`.

Methods:

- **`update`** –

  Index every bundle whose `manifest.json` moved since the last pass — all of them when `rebuild` — and return the rows appended.
- **`record`** –

  Append one row for a run the driver just wrote, so the index is current without waiting for a scan.
- **`query`** –

  A fluent query over the whole index — every runs directory the root knows, not only this one — with the live claims one `with_live` away.

#### update

```python
update(*, rebuild: bool = False) -> int
```

Index every bundle whose `manifest.json` moved since the last pass — all of them when `rebuild` — and return the rows appended.

#### record

```python
record(manifest: RunManifest, sim_dir: Path, source: RunSource = DRIVER) -> None
```

Append one row for a run the driver just wrote, so the index is current without waiting for a scan.

Parameters:

- **`manifest`** (`RunManifest`) –

  The run's manifest.
- **`sim_dir`** (`Path`) –

  The bundle directory.
- **`source`** (`RunSource`, default: `DRIVER` ) –

  How the row was produced.

#### query

```python
query() -> RunQuery
```

A fluent query over the whole index — every runs directory the root knows, not only this one — with the live claims one `with_live` away.

### RunQuery

```python
RunQuery(lazy: LazyFrame, predicates: tuple[Expr, ...] = (), limit: int | None = None, live: bool = False)
```

A fluent filter over the run index; each method returns a new query and nothing runs until `frame`.

Methods:

- **`where`** –

  A new query narrowed by `predicate`, judged on each run's latest row; the other filters are named shorthands for this.
- **`status`** –

  Runs in any of `statuses`.
- **`tags`** –

  Runs whose manifest metadata carries every one of these tag pairs.
- **`text`** –

  Runs whose id, description or notes contain `needle`, case-insensitively.
- **`since`** –

  Runs created at or after `when`.
- **`sweep`** –

  Runs solved under the sweep `sweep_id`.
- **`newest`** –

  Keep only the `count` most recently created runs, applied after the duplicate rows collapse.
- **`with_live`** –

  Also carry the lock-owner columns, so a listing can mark which runs are in flight.
- **`frame`** –

  The matching rows, newest first, one per run (the latest row wins), joined with the live claims when asked.
- **`run_ids`** –

  The ids of the matching runs, newest first.
- **`runs`** –

  The matching bundles as `Run` facades, skipping the ones retention has since removed.

Attributes:

- **`lazy`** (`LazyFrame`) –

  The run index rows the query filters.
- **`predicates`** (`tuple[Expr, ...]`) –

  The filters each run's latest row must pass.
- **`limit`** (`int | None`) –

  How many of the newest runs to keep; None keeps them all.
- **`live`** (`bool`) –

  Whether `frame` joins the lock-owner columns.

#### lazy

```python
lazy: LazyFrame
```

The run index rows the query filters.

#### predicates

```python
predicates: tuple[Expr, ...] = ()
```

The filters each run's latest row must pass.

#### limit

```python
limit: int | None = None
```

How many of the newest runs to keep; None keeps them all.

#### live

```python
live: bool = False
```

Whether `frame` joins the lock-owner columns.

#### where

```python
where(predicate: Expr) -> RunQuery
```

A new query narrowed by `predicate`, judged on each run's latest row; the other filters are named shorthands for this.

#### status

```python
status(*statuses: str) -> RunQuery
```

Runs in any of `statuses`.

#### tags

```python
tags(**pairs: str) -> RunQuery
```

Runs whose manifest metadata carries every one of these tag pairs.

#### text

```python
text(needle: str) -> RunQuery
```

Runs whose id, description or notes contain `needle`, case-insensitively.

#### since

```python
since(when: datetime) -> RunQuery
```

Runs created at or after `when`.

#### sweep

```python
sweep(sweep_id: str) -> RunQuery
```

Runs solved under the sweep `sweep_id`.

#### newest

```python
newest(count: int) -> RunQuery
```

Keep only the `count` most recently created runs, applied after the duplicate rows collapse.

#### with\_live

```python
with_live() -> RunQuery
```

Also carry the lock-owner columns, so a listing can mark which runs are in flight.

#### frame

```python
frame() -> DataFrame
```

The matching rows, newest first, one per run (the latest row wins), joined with the live claims when asked.

#### run\_ids

```python
run_ids() -> list[str]
```

The ids of the matching runs, newest first.

#### runs

```python
runs() -> list[Run]
```

The matching bundles as `Run` facades, skipping the ones retention has since removed.

### ArtifactColumn

Bases: `StrEnum`

The artifact index's columns: one row per stored artifact, keyed by `key`.

Attributes:

- **`KIND`** –

  The artifact kind name; the partition key.
- **`KEY`** –

  The content key the artifact is stored under.
- **`NAME`** –

  The `<adjective>-<noun>-<hash8>` store directory name.
- **`STORE_ROOT`** –

  The root of the store holding the artifact.
- **`CREATED_AT`** –

  When the artifact was published.
- **`SPEC_TYPE`** –

  `module:QualName` of the spec model.
- **`PRODUCER_VERSION`** –

  The kind's producer version at publish time.
- **`PRODUCER_HOST`** –

  The machine that produced the artifact.
- **`PRODUCER_PID`** –

  The id of the process that produced the artifact.
- **`FILE_COUNT`** –

  How many files the artifact holds.
- **`TOTAL_BYTES`** –

  The total size of the artifact's files in bytes.
- **`FILES`** –

  The artifact-relative names of its files.
- **`SPEC`** –

  The hashed spec payload, as JSON.
- **`FEMFORGE`** –

  The femforge release that produced the artifact.
- **`CODE_SHA`** –

  The producing checkout's commit.
- **`GMSH`** –

  The gmsh version, when meshing was involved.
- **`ELMER`** –

  The ElmerSolver version, when a solve was involved.
- **`DIR_MTIME`** –

  The artifact directory's modification time when the row was read.
- **`INDEXED_AT`** –

  When the row was written; the latest row per artifact wins.

#### KIND

```python
KIND = 'kind'
```

The artifact kind name; the partition key.

#### KEY

```python
KEY = 'key'
```

The content key the artifact is stored under.

#### NAME

```python
NAME = 'name'
```

The `<adjective>-<noun>-<hash8>` store directory name.

#### STORE\_ROOT

```python
STORE_ROOT = 'store_root'
```

The root of the store holding the artifact.

#### CREATED\_AT

```python
CREATED_AT = 'created_at'
```

When the artifact was published.

#### SPEC\_TYPE

```python
SPEC_TYPE = 'spec_type'
```

`module:QualName` of the spec model.

#### PRODUCER\_VERSION

```python
PRODUCER_VERSION = 'producer_version'
```

The kind's producer version at publish time.

#### PRODUCER\_HOST

```python
PRODUCER_HOST = 'producer_host'
```

The machine that produced the artifact.

#### PRODUCER\_PID

```python
PRODUCER_PID = 'producer_pid'
```

The id of the process that produced the artifact.

#### FILE\_COUNT

```python
FILE_COUNT = 'file_count'
```

How many files the artifact holds.

#### TOTAL\_BYTES

```python
TOTAL_BYTES = 'total_bytes'
```

The total size of the artifact's files in bytes.

#### FILES

```python
FILES = 'files'
```

The artifact-relative names of its files.

#### SPEC

```python
SPEC = 'spec'
```

The hashed spec payload, as JSON.

#### FEMFORGE

```python
FEMFORGE = 'femforge'
```

The femforge release that produced the artifact.

#### CODE\_SHA

```python
CODE_SHA = 'code_sha'
```

The producing checkout's commit.

#### GMSH

```python
GMSH = 'gmsh'
```

The gmsh version, when meshing was involved.

#### ELMER

```python
ELMER = 'elmer'
```

The ElmerSolver version, when a solve was involved.

#### DIR\_MTIME

```python
DIR_MTIME = 'dir_mtime'
```

The artifact directory's modification time when the row was read.

#### INDEXED\_AT

```python
INDEXED_AT = 'indexed_at'
```

When the row was written; the latest row per artifact wins.

### ConvergenceColumn

Bases: `StrEnum`

The keys `elmer.runtime.solve` writes into `convergence.jsonl`, one sample per solver iteration.

Attributes:

- **`WALL_TIME`** –

  `t_wall` `[s]`: Time since the parser started reading the solver output.
- **`SOLVER`** –

  The sif `Equation` name Elmer tags the sample with, lowercased.
- **`LOOP`** –

  Which loop the sample belongs to.
- **`ITER`** –

  One-based iteration count within the loop.
- **`NRM`** –

  `‖u‖₂` `[?]`: The solution's L₂ norm at this iteration.
- **`RELC`** –

  `Δ_rel` `[-]`: Relative change of the norm from the previous iteration.

#### WALL\_TIME

```python
WALL_TIME = 'wall_time'
```

`t_wall` `[s]`: Time since the parser started reading the solver output.

#### SOLVER

```python
SOLVER = 'solver'
```

The sif `Equation` name Elmer tags the sample with, lowercased.

#### LOOP

```python
LOOP = 'loop'
```

Which loop the sample belongs to.

#### ITER

```python
ITER = 'iter'
```

One-based iteration count within the loop.

#### NRM

```python
NRM = 'nrm'
```

`‖u‖₂` `[?]`: The solution's L₂ norm at this iteration.

#### RELC

```python
RELC = 'relc'
```

`Δ_rel` `[-]`: Relative change of the norm from the previous iteration.

### ConvergenceLoop

Bases: `StrEnum`

Which Elmer iteration loop a convergence sample belongs to: nonlinear or steady state.

Attributes:

- **`NS`** –

  The nonlinear inner loop.
- **`SS`** –

  The steady-state outer loop.

#### NS

```python
NS = 'NS'
```

The nonlinear inner loop.

#### SS

```python
SS = 'SS'
```

The steady-state outer loop.

### LiveRunColumn

Bases: `StrEnum`

The columns of the in-flight run table built from the run lock owners; never persisted.

Attributes:

- **`RUN_ID`** –

  The run the claim is held on.
- **`HOST`** –

  The machine holding the claim.
- **`PID`** –

  The id of the process holding the claim.
- **`STARTED_AT`** –

  When the claim was taken.
- **`HEARTBEAT_AT`** –

  When the holder last refreshed the claim.
- **`HEARTBEAT_AGE_S`** –

  `t_beat` `[s]`: Time since the holder last refreshed the claim.
- **`STALE`** –

  Whether the claim has gone without a heartbeat long enough for the next claimant to break it.

#### RUN\_ID

```python
RUN_ID = 'run_id'
```

The run the claim is held on.

#### HOST

```python
HOST = 'host'
```

The machine holding the claim.

#### PID

```python
PID = 'pid'
```

The id of the process holding the claim.

#### STARTED\_AT

```python
STARTED_AT = 'started_at'
```

When the claim was taken.

#### HEARTBEAT\_AT

```python
HEARTBEAT_AT = 'heartbeat_at'
```

When the holder last refreshed the claim.

#### HEARTBEAT\_AGE\_S

```python
HEARTBEAT_AGE_S = 'heartbeat_age_s'
```

`t_beat` `[s]`: Time since the holder last refreshed the claim.

#### STALE

```python
STALE = 'stale'
```

Whether the claim has gone without a heartbeat long enough for the next claimant to break it.

### LogColumn

Bases: `StrEnum`

The log table's columns; the sink emits these names and every query selects by them.

Attributes:

- **`DATE`** –

  The UTC day the record was logged on; a partition key.
- **`PROCESS`** –

  The sanitized name of the process that wrote the record; a partition key.
- **`TIME`** –

  When the record was logged, in UTC.
- **`LEVEL`** –

  The record's level name, such as `INFO`.
- **`LEVEL_NO`** –

  The record's numeric severity.
- **`NAME`** –

  The module that logged the record.
- **`FUNCTION`** –

  The function that logged the record.
- **`LINE`** –

  The source line that logged the record.
- **`MESSAGE`** –

  The formatted message.
- **`RUN_ID`** –

  The run the record was bound to; null outside a run.
- **`SWEEP_ID`** –

  The sweep the record was bound to; null outside a sweep.
- **`POINT_KEY`** –

  The sweep point the record was bound to; null outside a sweep.
- **`HOST`** –

  The machine that wrote the record.
- **`PID`** –

  The id of the process that wrote the record.
- **`THREAD`** –

  The name of the thread that logged the record.
- **`EXCEPTION`** –

  The formatted traceback of the exception the record carried; null when it carried none.
- **`EXTRA`** –

  Every other bound key as JSON; null when there are none.

#### DATE

```python
DATE = 'date'
```

The UTC day the record was logged on; a partition key.

#### PROCESS

```python
PROCESS = 'process'
```

The sanitized name of the process that wrote the record; a partition key.

#### TIME

```python
TIME = 'time'
```

When the record was logged, in UTC.

#### LEVEL

```python
LEVEL = 'level'
```

The record's level name, such as `INFO`.

#### LEVEL\_NO

```python
LEVEL_NO = 'level_no'
```

The record's numeric severity.

#### NAME

```python
NAME = 'name'
```

The module that logged the record.

#### FUNCTION

```python
FUNCTION = 'function'
```

The function that logged the record.

#### LINE

```python
LINE = 'line'
```

The source line that logged the record.

#### MESSAGE

```python
MESSAGE = 'message'
```

The formatted message.

#### RUN\_ID

```python
RUN_ID = 'run_id'
```

The run the record was bound to; null outside a run.

#### SWEEP\_ID

```python
SWEEP_ID = 'sweep_id'
```

The sweep the record was bound to; null outside a sweep.

#### POINT\_KEY

```python
POINT_KEY = 'point_key'
```

The sweep point the record was bound to; null outside a sweep.

#### HOST

```python
HOST = 'host'
```

The machine that wrote the record.

#### PID

```python
PID = 'pid'
```

The id of the process that wrote the record.

#### THREAD

```python
THREAD = 'thread'
```

The name of the thread that logged the record.

#### EXCEPTION

```python
EXCEPTION = 'exception'
```

The formatted traceback of the exception the record carried; null when it carried none.

#### EXTRA

```python
EXTRA = 'extra'
```

Every other bound key as JSON; null when there are none.

### RunColumn

Bases: `StrEnum`

The run index's columns: one row per run bundle, keyed by `run_id`.

Attributes:

- **`MONTH`** –

  The first day of the month the run was created in; the partition key.
- **`RUN_ID`** –

  The run's bundle directory name.
- **`CACHE_KEY`** –

  The full content hash of the run's result-affecting inputs.
- **`RUNS_DIR`** –

  The runs directory holding the bundle.
- **`STATUS`** –

  The manifest's lifecycle status.
- **`CREATED_AT`** –

  When the run was assembled.
- **`FINISHED_AT`** –

  When the run's latest stage finished; null before any has.
- **`CURRENT_STAGE`** –

  The stage running when the row was read; null when none was.
- **`STAGES_DONE`** –

  How many stages had finished when the row was read.
- **`GIT_SHA`** –

  The commit of the checkout the run was made in, with `-dirty` when it had uncommitted changes.
- **`ELMER_VERSION`** –

  The ElmerSolver version parsed from the solver log.
- **`PHYSICS`** –

  The physics preset the run solved.
- **`ELEMENT_COUNT`** –

  Total number of volume elements in the solved mesh.
- **`MIN_ELEMENT_SIZE`** –

  `h_min` `[m]`: Smallest element edge length.
- **`MAX_ELEMENT_SIZE`** –

  `h_max` `[m]`: Largest element edge length.
- **`DESCRIPTION`** –

  The description in the run's metadata.
- **`TAGS`** –

  The tags in the run's metadata, as a JSON object.
- **`NOTES`** –

  The manifest's free-form provenance note.
- **`MESH_NAME`** –

  The mesh artifact's store directory name.
- **`MESH_KEY`** –

  The mesh artifact's content key.
- **`PUBLISHED`** –

  Whether the bundle was published to the artifact store.
- **`HAS_RESULT`** –

  Whether the bundle holds a readable `result.json`.
- **`HAS_VTU`** –

  Whether the bundle holds a result VTU file.
- **`CLEANUPS`** –

  How many retention passes the manifest records.
- **`BUNDLE_BYTES`** –

  The bundle directory's total size in bytes.
- **`SWEEP_ID`** –

  The sweep the run was solved under; null outside a sweep.
- **`POINT_KEY`** –

  The sweep point the run was solved for; null outside a sweep.
- **`MANIFEST_MTIME`** –

  The manifest's modification time when the row was read.
- **`INDEXED_AT`** –

  When the row was written; the latest row per run wins.
- **`SOURCE`** –

  How the row was produced.
- **`MODEL_DIM`** –

  The mesh dimension the run's frame solves on: 2 or 3.
- **`ARTIFACT_KEYS`** –

  The content keys of the store artifacts the run consumed.
- **`FORCE_Y`** –

  `F_y` `[N]`: Y component of the net force on the moving array.
- **`TEMPERATURE_MAX`** –

  `T_max` `[K]`: Peak temperature in the domain.
- **`WALL_TIME`** –

  `t_wall` `[s]`: Solver wall-clock time.

#### MONTH

```python
MONTH = 'month'
```

The first day of the month the run was created in; the partition key.

#### RUN\_ID

```python
RUN_ID = 'run_id'
```

The run's bundle directory name.

#### CACHE\_KEY

```python
CACHE_KEY = 'cache_key'
```

The full content hash of the run's result-affecting inputs.

#### RUNS\_DIR

```python
RUNS_DIR = 'runs_dir'
```

The runs directory holding the bundle.

#### STATUS

```python
STATUS = 'status'
```

The manifest's lifecycle status.

#### CREATED\_AT

```python
CREATED_AT = 'created_at'
```

When the run was assembled.

#### FINISHED\_AT

```python
FINISHED_AT = 'finished_at'
```

When the run's latest stage finished; null before any has.

#### CURRENT\_STAGE

```python
CURRENT_STAGE = 'current_stage'
```

The stage running when the row was read; null when none was.

#### STAGES\_DONE

```python
STAGES_DONE = 'stages_done'
```

How many stages had finished when the row was read.

#### GIT\_SHA

```python
GIT_SHA = 'git_sha'
```

The commit of the checkout the run was made in, with `-dirty` when it had uncommitted changes.

#### ELMER\_VERSION

```python
ELMER_VERSION = 'elmer_version'
```

The ElmerSolver version parsed from the solver log.

#### PHYSICS

```python
PHYSICS = 'physics'
```

The physics preset the run solved.

#### ELEMENT\_COUNT

```python
ELEMENT_COUNT = 'element_count'
```

Total number of volume elements in the solved mesh.

#### MIN\_ELEMENT\_SIZE

```python
MIN_ELEMENT_SIZE = 'min_element_size'
```

`h_min` `[m]`: Smallest element edge length.

#### MAX\_ELEMENT\_SIZE

```python
MAX_ELEMENT_SIZE = 'max_element_size'
```

`h_max` `[m]`: Largest element edge length.

#### DESCRIPTION

```python
DESCRIPTION = 'description'
```

The description in the run's metadata.

#### TAGS

```python
TAGS = 'tags'
```

The tags in the run's metadata, as a JSON object.

#### NOTES

```python
NOTES = 'notes'
```

The manifest's free-form provenance note.

#### MESH\_NAME

```python
MESH_NAME = 'mesh_name'
```

The mesh artifact's store directory name.

#### MESH\_KEY

```python
MESH_KEY = 'mesh_key'
```

The mesh artifact's content key.

#### PUBLISHED

```python
PUBLISHED = 'published'
```

Whether the bundle was published to the artifact store.

#### HAS\_RESULT

```python
HAS_RESULT = 'has_result'
```

Whether the bundle holds a readable `result.json`.

#### HAS\_VTU

```python
HAS_VTU = 'has_vtu'
```

Whether the bundle holds a result VTU file.

#### CLEANUPS

```python
CLEANUPS = 'cleanups'
```

How many retention passes the manifest records.

#### BUNDLE\_BYTES

```python
BUNDLE_BYTES = 'bundle_bytes'
```

The bundle directory's total size in bytes.

#### SWEEP\_ID

```python
SWEEP_ID = 'sweep_id'
```

The sweep the run was solved under; null outside a sweep.

#### POINT\_KEY

```python
POINT_KEY = 'point_key'
```

The sweep point the run was solved for; null outside a sweep.

#### MANIFEST\_MTIME

```python
MANIFEST_MTIME = 'manifest_mtime'
```

The manifest's modification time when the row was read.

#### INDEXED\_AT

```python
INDEXED_AT = 'indexed_at'
```

When the row was written; the latest row per run wins.

#### SOURCE

```python
SOURCE = 'source'
```

How the row was produced.

#### MODEL\_DIM

```python
MODEL_DIM = 'model_dim'
```

The mesh dimension the run's frame solves on: 2 or 3.

#### ARTIFACT\_KEYS

```python
ARTIFACT_KEYS = 'artifact_keys'
```

The content keys of the store artifacts the run consumed.

#### FORCE\_Y

```python
FORCE_Y = 'force_y'
```

`F_y` `[N]`: Y component of the net force on the moving array.

#### TEMPERATURE\_MAX

```python
TEMPERATURE_MAX = 'temperature_max'
```

`T_max` `[K]`: Peak temperature in the domain.

#### WALL\_TIME

```python
WALL_TIME = 'wall_time'
```

`t_wall` `[s]`: Solver wall-clock time.

### RunSource

Bases: `StrEnum`

How a run index row was produced: a filesystem scan of the bundles, or the Driver reporting live.

Attributes:

- **`SCAN`** –

  Read from the bundle on disk by an index pass.
- **`DRIVER`** –

  Recorded by the Driver as it wrote the bundle.

#### SCAN

```python
SCAN = 'scan'
```

Read from the bundle on disk by an index pass.

#### DRIVER

```python
DRIVER = 'driver'
```

Recorded by the Driver as it wrote the bundle.

### SweepColumn

Bases: `StrEnum`

The sweep table's columns: one row per sweep launch.

Attributes:

- **`MONTH`** –

  The first day of the month the sweep started in; the partition key.
- **`SWEEP_ID`** –

  The sweep's id.
- **`DESCRIPTION`** –

  What the sweep runs, as its launcher described it.
- **`STARTED_AT`** –

  When the sweep started.
- **`FINISHED_AT`** –

  When the sweep finished; null on its launch row.
- **`TOTAL`** –

  How many points the sweep holds, skipped ones included.
- **`SKIPPED`** –

  How many points were already stored and did not run.
- **`MAX_WORKERS`** –

  How many worker processes run the points.
- **`HOST`** –

  The machine that launched the sweep.
- **`PID`** –

  The id of the process that launched the sweep.
- **`PROCESS`** –

  The sanitized name of the process that launched the sweep.
- **`GIT_SHA`** –

  The commit of the checkout the sweep ran in.
- **`INDEXED_AT`** –

  When the row was written; the closing row supersedes the launch row.

#### MONTH

```python
MONTH = 'month'
```

The first day of the month the sweep started in; the partition key.

#### SWEEP\_ID

```python
SWEEP_ID = 'sweep_id'
```

The sweep's id.

#### DESCRIPTION

```python
DESCRIPTION = 'description'
```

What the sweep runs, as its launcher described it.

#### STARTED\_AT

```python
STARTED_AT = 'started_at'
```

When the sweep started.

#### FINISHED\_AT

```python
FINISHED_AT = 'finished_at'
```

When the sweep finished; null on its launch row.

#### TOTAL

```python
TOTAL = 'total'
```

How many points the sweep holds, skipped ones included.

#### SKIPPED

```python
SKIPPED = 'skipped'
```

How many points were already stored and did not run.

#### MAX\_WORKERS

```python
MAX_WORKERS = 'max_workers'
```

How many worker processes run the points.

#### HOST

```python
HOST = 'host'
```

The machine that launched the sweep.

#### PID

```python
PID = 'pid'
```

The id of the process that launched the sweep.

#### PROCESS

```python
PROCESS = 'process'
```

The sanitized name of the process that launched the sweep.

#### GIT\_SHA

```python
GIT_SHA = 'git_sha'
```

The commit of the checkout the sweep ran in.

#### INDEXED\_AT

```python
INDEXED_AT = 'indexed_at'
```

When the row was written; the closing row supersedes the launch row.

### SweepEventColumn

Bases: `StrEnum`

The sweep event table's columns: one row per point state transition.

Attributes:

- **`SWEEP_ID`** –

  The sweep the point belongs to; the partition key.
- **`POINT_KEY`** –

  The point's key within the sweep.
- **`POINT_NAME`** –

  A readable label for the point.
- **`STATE`** –

  The state the point entered.
- **`AT`** –

  When the point entered the state.
- **`HOST`** –

  The machine that recorded the transition.
- **`PID`** –

  The id of the process that recorded the transition.
- **`ORIGIN`** –

  Where a finished point's result came from, such as an artifact origin.
- **`ERROR`** –

  The traceback a failed point recorded.
- **`DURATION_S`** –

  `t_point` `[s]`: How long the point took up to this transition.

#### SWEEP\_ID

```python
SWEEP_ID = 'sweep_id'
```

The sweep the point belongs to; the partition key.

#### POINT\_KEY

```python
POINT_KEY = 'point_key'
```

The point's key within the sweep.

#### POINT\_NAME

```python
POINT_NAME = 'point_name'
```

A readable label for the point.

#### STATE

```python
STATE = 'state'
```

The state the point entered.

#### AT

```python
AT = 'at'
```

When the point entered the state.

#### HOST

```python
HOST = 'host'
```

The machine that recorded the transition.

#### PID

```python
PID = 'pid'
```

The id of the process that recorded the transition.

#### ORIGIN

```python
ORIGIN = 'origin'
```

Where a finished point's result came from, such as an artifact origin.

#### ERROR

```python
ERROR = 'error'
```

The traceback a failed point recorded.

#### DURATION\_S

```python
DURATION_S = 'duration_s'
```

`t_point` `[s]`: How long the point took up to this transition.

### SweepState

Bases: `StrEnum`

The lifecycle states a sweep point moves through; each transition is one sweep event row.

Attributes:

- **`QUEUED`** –

  Waiting for a worker.
- **`RUNNING`** –

  Being solved by a worker.
- **`DONE`** –

  Finished successfully.
- **`FAILED`** –

  Finished with an error; its event carries the traceback.
- **`SKIPPED`** –

  Already stored when the sweep started, so never queued.

#### QUEUED

```python
QUEUED = 'queued'
```

Waiting for a worker.

#### RUNNING

```python
RUNNING = 'running'
```

Being solved by a worker.

#### DONE

```python
DONE = 'done'
```

Finished successfully.

#### FAILED

```python
FAILED = 'failed'
```

Finished with an error; its event carries the traceback.

#### SKIPPED

```python
SKIPPED = 'skipped'
```

Already stored when the sweep started, so never queued.

### LogQuery

```python
LogQuery(frame: LazyFrame)
```

A fluent filter over the log frame; each method returns a new query and nothing runs until `page`.

Methods:

- **`over`** –

  A query spanning one telemetry root's parquet shards and live journals.
- **`where`** –

  A new query narrowed by `predicate`; the other methods are named shorthands for this.
- **`levels`** –

  Records at any of `levels`.
- **`processes`** –

  Records written by any of the processes `names`.
- **`run`** –

  Records bound to the run `run_id`.
- **`sweep`** –

  Records bound to the sweep `sweep_id`.
- **`since`** –

  Records logged at or after `when`.
- **`until`** –

  Records logged at or before `when`.
- **`text`** –

  Records whose message contains `needle`, case-sensitively.
- **`page`** –

  Collect one slice of the result, oldest record first.
- **`tail`** –

  Collect the newest `limit` records, oldest of them first.

#### over

```python
over(telemetry: Telemetry) -> Self
```

A query spanning one telemetry root's parquet shards and live journals.

#### where

```python
where(predicate: Expr) -> Self
```

A new query narrowed by `predicate`; the other methods are named shorthands for this.

#### levels

```python
levels(*levels: LogLevel | str) -> Self
```

Records at any of `levels`.

#### processes

```python
processes(*names: str) -> Self
```

Records written by any of the processes `names`.

#### run

```python
run(run_id: str) -> Self
```

Records bound to the run `run_id`.

#### sweep

```python
sweep(sweep_id: str) -> Self
```

Records bound to the sweep `sweep_id`.

#### since

```python
since(when: datetime) -> Self
```

Records logged at or after `when`.

#### until

```python
until(when: datetime) -> Self
```

Records logged at or before `when`.

#### text

```python
text(needle: str) -> Self
```

Records whose message contains `needle`, case-sensitively.

#### page

```python
page(offset: int = 0, limit: int = 200) -> DataFrame
```

Collect one slice of the result, oldest record first.

#### tail

```python
tail(limit: int = 200) -> DataFrame
```

Collect the newest `limit` records, oldest of them first.

### ParquetLogSink

```python
ParquetLogSink(telemetry: Telemetry, *, process: str | None = None, flush_every: int = FLUSH_EVERY, flush_interval_s: float = FLUSH_INTERVAL_S)
```

Loguru sink writing every record to a JSONL journal at once and to hive-partitioned parquet in batches.

Parameters:

- **`telemetry`** (`Telemetry`) –

  The telemetry root the records land under.
- **`process`** (`str | None`, default: `None` ) –

  The process name the records are partitioned under; None uses the script's name.
- **`flush_every`** (`int`, default: `FLUSH_EVERY` ) –

  How many records trigger a checkpoint into parquet.
- **`flush_interval_s`** (`float`, default: `FLUSH_INTERVAL_S` ) –

  Seconds between timed checkpoints.

Methods:

- **`__call__`** –

  The loguru sink protocol; add this callable with `format="{message}"` at DEBUG.
- **`flush`** –

  Checkpoint the journal's records into the parquet store; a no-op when nothing is pending.
- **`close`** –

  Stop the timer, checkpoint whatever is left and drop the journal; idempotent.

#### \_\_call\_\_

```python
__call__(message: Message) -> None
```

The loguru sink protocol; add this callable with `format="{message}"` at DEBUG.

#### flush

```python
flush() -> None
```

Checkpoint the journal's records into the parquet store; a no-op when nothing is pending.

#### close

```python
close() -> None
```

Stop the timer, checkpoint whatever is left and drop the journal; idempotent.

### SweepJournal

```python
SweepJournal(telemetry: Telemetry, *, sweep_id: str, process: str | None = None)
```

One sweep's writer: the launch and closing rows in the sweeps table, every point transition through this process's own journal.

Parameters:

- **`telemetry`** (`Telemetry`) –

  The telemetry root the sweep is recorded under.
- **`sweep_id`** (`str`) –

  The sweep this writer records.
- **`process`** (`str | None`, default: `None` ) –

  The process name stamped on the sweep row; None uses the script's name.

Methods:

- **`start`** –

  Open the sweep: stamp its launch row so a reader sees it the moment the first point starts.
- **`event`** –

  Journal one point transition; the parent and each worker write through a journal of their own.
- **`checkpoint`** –

  Fold this process's journalled events into parquet without closing the sweep.
- **`finish`** –

  Close the sweep: the row gains a finish time, every journal lands in parquet, and the shards fold into one per sweep.

#### start

```python
start(description: str, total: int, skipped: int = 0, max_workers: int = 1) -> None
```

Open the sweep: stamp its launch row so a reader sees it the moment the first point starts.

Parameters:

- **`description`** (`str`) –

  What the sweep runs.
- **`total`** (`int`) –

  How many points the sweep holds, skipped ones included.
- **`skipped`** (`int`, default: `0` ) –

  How many points were already stored and will not run.
- **`max_workers`** (`int`, default: `1` ) –

  How many worker processes run the points.

#### event

```python
event(point_key: str, state: SweepState, *, point_name: str | None = None, origin: str | None = None, error: str | None = None, duration_s: float | None = None) -> None
```

Journal one point transition; the parent and each worker write through a journal of their own.

Parameters:

- **`point_key`** (`str`) –

  The point's key within the sweep.
- **`state`** (`SweepState`) –

  The state the point enters.
- **`point_name`** (`str | None`, default: `None` ) –

  A readable label for the point.
- **`origin`** (`str | None`, default: `None` ) –

  Where a finished point's result came from, such as an artifact origin.
- **`error`** (`str | None`, default: `None` ) –

  The traceback of a failed point.
- **`duration_s`** (`float | None`, default: `None` ) –

  Seconds the point took up to this transition.

#### checkpoint

```python
checkpoint() -> None
```

Fold this process's journalled events into parquet without closing the sweep.

#### finish

```python
finish() -> None
```

Close the sweep: the row gains a finish time, every journal lands in parquet, and the shards fold into one per sweep.

### SweepQuery

```python
SweepQuery(telemetry: Telemetry, sweep_id: str)
```

One sweep read back: its launch row, every point transition (settled shards plus live journals), and the progress summary.

Methods:

- **`launch`** –

  The sweep's own row: its closing one when it has finished, else the one its launch wrote; empty when unknown.
- **`events`** –

  Every transition of this sweep's points, oldest first, unioning what is still only in the live journals.
- **`latest`** –

  The state each point is in now: its last transition, one row per `point_key`.
- **`failures`** –

  The points whose last transition failed, each carrying the traceback the worker recorded.
- **`summary`** –

  Where the sweep stands at `now`: the per-state counts, the trailing-window rate, and the ETA that rate implies.

#### launch

```python
launch() -> DataFrame
```

The sweep's own row: its closing one when it has finished, else the one its launch wrote; empty when unknown.

#### events

```python
events() -> DataFrame
```

Every transition of this sweep's points, oldest first, unioning what is still only in the live journals.

#### latest

```python
latest() -> DataFrame
```

The state each point is in now: its last transition, one row per `point_key`.

#### failures

```python
failures() -> DataFrame
```

The points whose last transition failed, each carrying the traceback the worker recorded.

#### summary

```python
summary(now: datetime | None = None) -> SweepSummary
```

Where the sweep stands at `now`: the per-state counts, the trailing-window rate, and the ETA that rate implies.

### SweepSummary

```python
SweepSummary(queued: int, running: int, done: int, failed: int, skipped: int, rate_per_h: float, eta_s: float | None, elapsed_s: float)
```

Where a sweep stands: how many points sit in each state, how fast they are landing, and how long the rest should take.

Attributes:

- **`queued`** (`int`) –

  Points waiting for a worker.
- **`running`** (`int`) –

  Points a worker is solving.
- **`done`** (`int`) –

  Points that finished successfully.
- **`failed`** (`int`) –

  Points that finished with an error.
- **`skipped`** (`int`) –

  Points already stored when the sweep started.
- **`rate_per_h`** (`float`) –

  `ṅ` `[1/h]`: Points settled per hour over the trailing window.
- **`eta_s`** (`float | None`) –

  `t_eta` `[s]`: Seconds the outstanding points should take at that rate; None while nothing has settled.
- **`elapsed_s`** (`float`) –

  `t_elapsed` `[s]`: Seconds since the sweep started, up to its finish when it has one.

#### queued

```python
queued: int
```

Points waiting for a worker.

#### running

```python
running: int
```

Points a worker is solving.

#### done

```python
done: int
```

Points that finished successfully.

#### failed

```python
failed: int
```

Points that finished with an error.

#### skipped

```python
skipped: int
```

Points already stored when the sweep started.

#### rate\_per\_h

```python
rate_per_h: float
```

`ṅ` `[1/h]`: Points settled per hour over the trailing window.

#### eta\_s

```python
eta_s: float | None
```

`t_eta` `[s]`: Seconds the outstanding points should take at that rate; None while nothing has settled.

#### elapsed\_s

```python
elapsed_s: float
```

`t_elapsed` `[s]`: Seconds since the sweep started, up to its finish when it has one.

### TextTail

```python
TextTail(path: Path)
```

A byte-offset cursor over one growing text file; a truncation rewinds it and an unterminated last line is held back until it ends.

Methods:

- **`reset`** –

  Forget the cursor so the next read starts from the beginning of the file.
- **`read_new`** –

  Every complete line written since the last call, newline stripped; empty when the file is absent or unchanged.

#### reset

```python
reset() -> None
```

Forget the cursor so the next read starts from the beginning of the file.

#### read\_new

```python
read_new() -> list[str]
```

Every complete line written since the last call, newline stripped; empty when the file is absent or unchanged.

### LiveLogTap

```python
LiveLogTap(*, process: str | None = None, maxlen: int = BUFFER_SIZE)
```

Loguru sink keeping the last records in memory and pushing each new one onto every asyncio subscriber queue.

Methods:

- **`__call__`** –

  The loguru sink protocol; buffer the record, then hand it to each subscriber's loop.
- **`snapshot`** –

  The buffered records, oldest first — what a subscriber renders before its first live record arrives.
- **`subscribe`** –

  Register a queue fed on `loop` and return it; the caller passes it back to `unsubscribe`.
- **`unsubscribe`** –

  Stop feeding `queue`; a queue that was never registered is ignored.

#### \_\_call\_\_

```python
__call__(message: Message) -> None
```

The loguru sink protocol; buffer the record, then hand it to each subscriber's loop.

#### snapshot

```python
snapshot() -> list[LogRow]
```

The buffered records, oldest first — what a subscriber renders before its first live record arrives.

#### subscribe

```python
subscribe(loop: AbstractEventLoop, *, maxsize: int = QUEUE_SIZE) -> Queue[LogRow]
```

Register a queue fed on `loop` and return it; the caller passes it back to `unsubscribe`.

#### unsubscribe

```python
unsubscribe(queue: Queue[LogRow]) -> None
```

Stop feeding `queue`; a queue that was never registered is ignored.

### current\_sweep\_point

```python
current_sweep_point() -> SweepPoint | None
```

The sweep point in scope, or None outside any `sweep_point` block.

### sweep\_point

```python
sweep_point(sweep_id: str, point_key: str) -> Iterator[SweepPoint]
```

Mark this context as one sweep point, binding both ids onto every log record emitted inside it.

### scan\_convergence

```python
scan_convergence(sim_dir: str | Path, *, with_progress: bool = True) -> DataFrame
```

A run's `convergence.jsonl` as a typed frame, carrying each loop's first relative change and how far it has come toward its tolerance.

### coerce

```python
coerce(frame: LazyFrame, schema: Schema) -> LazyFrame
```

Cast a loosely-typed frame (JSON text, a partial row set) onto `schema`, parsing date/time strings and filling missing columns with nulls.

### journal\_glob

```python
journal_glob(name: str) -> str
```

The glob matching every process's journal for the table `name`.

### recover\_journals

```python
recover_journals(live_dir: Path, store: HiveStore, pattern: str, *, alive: Callable[[int], bool] = pid_exists) -> list[Path]
```

Checkpoint and delete the journals of processes that are gone, so a crash costs no records; returns the journals recovered.

Parameters:

- **`live_dir`** (`Path`) –

  The directory holding the journals.
- **`store`** (`HiveStore`) –

  The table the recovered records are appended to.
- **`pattern`** (`str`) –

  The glob selecting that table's journals.
- **`alive`** (`Callable[[int], bool]`, default: `pid_exists` ) –

  Whether a process id is still running; the journal of a live writer is left alone.

### scan\_journals

```python
scan_journals(live_dir: Path, pattern: str, schema: Schema) -> LazyFrame
```

A frame over every matching journal's records, coerced onto `schema`; lines that are not JSON records are skipped.

Parameters:

- **`live_dir`** (`Path`) –

  The directory holding the journals.
- **`pattern`** (`str`) –

  The glob selecting the journals to read.
- **`schema`** (`Schema`) –

  The table schema the records are coerced onto.

### require\_root

```python
require_root(explicit: str | Path | None = None) -> Path
```

The telemetry root a reader needs, or `TelemetryRootMissing` naming both ways to provide one.

### resolve\_root

```python
resolve_root(explicit: str | Path | None = None) -> Path | None
```

The telemetry root: `explicit` if given, else `FEMFORGE_TELEMETRY`, else None because telemetry is off.

### live\_runs

```python
live_runs(runs_dir: str | Path) -> DataFrame
```

The runs claimed right now under `runs_dir`, read from each `<run_id>.lock/owner.json` with the age of its heartbeat.

### scan\_logs

```python
scan_logs(telemetry: Telemetry) -> LazyFrame
```

Every log record: the settled parquet shards unioned with what is still only in the live journals.

### sweep\_runs

```python
sweep_runs(telemetry: Telemetry) -> DataFrame
```

Every sweep the root knows, newest first and one row each (a finished sweep's closing row wins).