Skip to content
epftoolbox2

Incremental Training

When training models over long test periods, epftoolbox2 can save intermediate results and resume if interrupted.

Enabling Incremental Training

report = pipeline.run(
    data=df,
    test_start="2024-01-01",
    test_end="2024-12-31",
    save_dir="results",  # Enable incremental saving
)

How It Works

  1. Each task (date × hour × horizon × model) is saved after completion
  2. If the script is interrupted, re-running loads completed tasks
  3. Only missing tasks are computed

Cache Directory Structure

Each model gets a results file and a manifest next to it:

  • Directoryresults/
    • ols.jsonl
    • ols.manifest.json
    • lasso.jsonl
    • lasso.manifest.json

Results File Format

Each line in results.jsonl is a JSON object:

{"run_date": "2024-02-01", "target_date": "2024-02-02", "hour": 1, "horizon": 1, "day_in_test": 0, "prediction": 92.08884109589042, "actual": 58.0, "coefficients": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}
{"run_date": "2024-02-01", "target_date": "2024-02-02", "hour": 0, "horizon": 1, "day_in_test": 0, "prediction": 92.33908219178082, "actual": 58.62, "coefficients": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}

A task is considered already done by its (hour, horizon, run_date), the actual calendar date, not the day_in_test offset from test_start, so extending or shifting test_start between runs into the same directory can’t make an unrelated date look already computed.

The Manifest

<model>.manifest.json records what produced <model>.jsonl: the model class and its hyperparameters (all of them: whatever the model returns from its _model_kwargs), the predictors’ per-(period, horizon) order (not just which names are used: a reordering changes what stored coefficients mean, and a hash of the ordered layout tells ["price_d-1", "price_d-2"] apart from ["price_d-{horizon}"]), target, horizon, freq, training window, a content hash of the data columns used up to test_end (rows after test_end don’t affect any forecast in this run, so appending them between runs isn’t a mismatch), and the data’s own time zone (converting the same instants to a different zone keeps the same hash, since it’s over each value’s own representation, not the zone label): everything that defines what a forecast in the file actually is. It also records provenance (package version, git SHA if available, when it was first and last written) and the information set settings, but neither is compared: a version bump or a new commit does not by itself invalidate a results directory, and the information set gates which predictors are allowed before fitting rather than changing what a fitted model predicts.

Before resuming, the pipeline compares a manifest built from the current run to the one on disk. If they disagree, it raises rather than silently mixing forecasts made under two configurations:

ManifestMismatchError: OLS: the results in results were produced with a different configuration than this
run (ols.manifest.json):
  - predictor_layout_hash: '9c2c1f...' (existing results) vs 'a71e08...' (this run)
Resuming would silently mix forecasts made under two configurations. Use a different save_dir, remove the
existing results, or pass allow_manifest_mismatch=True (`epf model --allow-manifest-mismatch`) to overwrite
the manifest and resume anyway.
report = pipeline.run(
    data=df,
    test_start="2024-01-01",
    test_end="2024-12-31",
    save_dir="results",
    allow_manifest_mismatch=True,  # overwrite the manifest and resume anyway
)

A .jsonl file with no .manifest.json next to it is not a mismatch: by default it is accepted with a warning, and a manifest is written from then on. Pass require_manifest=True to refuse them instead, if you want the stricter guarantee for every results directory you resume into:

report = pipeline.run(..., save_dir="results", require_manifest=True)