Artifacts and retention¶

An artifact store keeps the expensive intermediate products of a run, the solve-frame STEP, the Elmer mesh and the finished bundle, under a key computed from the inputs that made them. A second run that needs the same mesh fetches it instead of meshing again, and a run already solved on another machine comes back without solving. Retention is the other half: a policy that says which bundle files leave after a run, so a sweep of thousands of runs does not fill the disk. Both are driver settings, so neither changes a run's identity.

Configure a driver with a store¶

import femforge as ff

store = ff.artifacts.LocalArtifactStore("artifact-store", link=True)

driver = ff.Driver(
    runs_dir="runs",
    artifacts=ff.artifacts.ArtifactCache(store),
    reuse=True,
    bundle_policy=ff.run.BundlePolicy.FIELDS,
    retention=ff.run.RetentionPolicy.lean(),
)

With link=True the store hardlinks files into the run directory where the volume allows, else it copies them. A hardlinked file shares its bytes with the store, so never edit a materialized file in place.

Kinds and keys¶

Each artifact belongs to a kind, a module-level ArtifactKind passed by reference. A kind names its spec model, the files a complete artifact holds, and a producer version that you bump to invalidate every existing key of the kind.

A key is the hex SHA-256 of the spec's JSON with sorted keys, after every field marked HashExclude() is removed at any depth. The mesh key leaves out conditions, material properties, physics and metadata, so two configs that differ only in a boundary condition share one mesh. A cached mesh is rebound to the current config's domain: materials by tag and tags by selector. A mismatch raises MeshRebindError, so a condition never lands on the wrong boundary.

An authored STEP under a Solid() frame passes through: the solve meshes the file itself, so no STEP artifact is cached for it.

The store¶

LocalArtifactStore keeps each artifact in a directory under the store root:

artifact-store/
├── step/
│   └── glad-dove-5d20e8b1/
│       ├── artifact.json
│       └── geometry.step
├── elmer_mesh/
│   ├── .staging/                          producer workdirs
│   ├── jolly-crow-91a3f6c2/
│   │   ├── artifact.json
│   │   └── mesh.header ...
│   └── kind-coot-07be44d9.lock/            a producer's claim
└── run_bundle/
    └── calm-avocet-3f9a1c07/
        ├── artifact.json
        └── manifest.json ...

The directory name is <adjective>-<noun>-<hash8>, a pure function of the key, so a key resolves to its directory without a scan. Every persisted key carries that name beside it: ArtifactRecord, ArtifactRef and the manifest's mesh_name all derive it from the key on load.

An artifact exists exactly when its artifact.json sidecar can be read. The sidecar lists every file with its size and SHA-256, the exact spec payload that was hashed, the producer's host and process id, and the tool versions (femforge, the commit, gmsh, Elmer). A publish fills a staging directory and renames it into place, so the first publisher wins and a second one discards its copy. A fetch always checks file sizes, and checks hashes on request. An unreadable sidecar is moved aside as <name>.corrupt-<stamp> so a fresh publish can land.

What the driver fetches and publishes¶

The manifest records each store-sourced input in artifacts and the bundle's own entry in published. That is enough to put the inputs back later (femforge artifacts rehydrate) and lets retention drop them freely.

A producer claims its key before it produces, refreshes the claim every 10 s, and another producer waits for it and serves its result. A claim silent for 120 s is stale and the next producer breaks it.

ArtifactCache and the remote tier¶

ArtifactCache always consults its local store first. Its optional remote tier takes any ArtifactStore: a hit there is pulled through into the local store, and every publish is pushed to it after the local commit.

With a remote tier, producers claim keys there, so machines that share it produce each artifact once. femforge ships the directory store only; a second LocalArtifactStore on a shared volume can serve as the remote tier.

The FEMFORGE_ARTIFACTS variable names a store root for the command line: the femforge artifacts commands use it when you pass no --root, and femforge doctor checks its free disk space. The driver never reads it; pass the cache to Driver(artifacts=...).

Bundle policy¶

BundlePolicy decides which bundle files a publish carries. It also decides reuse: under FIELDS or FULL, a local bundle whose field output was purged no longer counts as reusable.

Retention policy¶

RetentionPolicy holds one Cleanup rule per class of bundle file. NEVER keeps the files, ON_SUCCESS removes them when the run ended ok, and ALWAYS removes them whatever the status. manifest.json and result.json belong to no class a policy can name, so they stay.

Every class defaults to NEVER and cached_inputs to ALWAYS.

The driver applies its policy in the cleanup stage, and evaluate(retention=...) overrides it for one call. Every pass that removes something appends a CleanupRecord to the manifest's cleanups. Run.purged(file_class) reads them back, and Run.mesh() names the pass when the VTU is gone.

Purge existing bundles¶

femforge purge applies a policy to every bundle under a runs directory after the fact. It is a dry run unless you pass --apply:

femforge purge --runs-dir runs --preset lean
femforge purge --runs-dir runs --preset lean --fields always --only-status ok --older-than 30 --apply

The dry run prints the files and gigabytes per class it would free. Each field of the policy has an override flag, such as --fields or --cached-inputs, taking never, on_success or always. --only-status limits the purge to bundles in the given statuses and --older-than to bundles created at least that many days ago. The purge skips links, directories that are not bundles, unreadable manifests, and running bundles whose manifest was written within --running-grace hours (6 by default).

Browse the store¶

femforge artifacts lists and inspects a local store: ls, show, path, materialize and rehydrate. A reference is a full key, a hex prefix of 8 or more characters, a <adjective>-<noun>-<hash8> name, an artifact directory, or a path inside one. The store root is --root, else FEMFORGE_ARTIFACTS, else the store a path argument lies inside. rehydrate fetches a bundle's store-sourced inputs back beside it. See CLI for every option.