Data Caching

epftoolbox2 caches downloaded data to avoid redundant API calls and speed up subsequent runs. There are two independent caches:

• Source cache (cache=True): one cache per source under .cache/sources/, filled range by range. Recommended.
• Single-file cache (cache="file.csv", cache="file.parquet" or cache="some/dir/"): the combined output of all sources for one exact configuration and date range.

Both return exactly the same frame as cache=False: every source’s output is clipped to the requested half-open range [start, end) whether it comes from the API or from disk.

# Source-level caching (recommended)
df = pipeline.run(start="2024-01-01", end="2024-06-01", cache=True)

# Whole source output in one file (CSV, or Parquet for *.parquet)
df = pipeline.run(start="2024-01-01", end="2024-06-01", cache="my_data.csv")

# A directory: one Parquet file per configuration and date range
df = pipeline.run(start="2024-01-01", end="2024-06-01", cache="$SCRATCH/epf_cache/")

Source cache (cache=True)

1. Look up the key - each source gets a directory named after a hash of the cache format version, the source class and source.get_cache_config()

2. Find missing ranges - metadata.json lists the ranges already cached. Only the gaps are fetched

3. Fetch and record - the fetched data is stored, but only the part the source actually returned, and never the last two days before now (see below)

4. Combine - each cached file is read only inside its own recorded range, then cached and fresh pieces are joined in time order

• Directory.cache/

  • Directorysources/

    • Directorya1b2c3d4e5f6a7b8/ (EntsoeSource)

      • metadata.json
      • data_20230101T000000_20240101T000000.parquet
      • data_20240101T000000_20240601T000000.parquet
    • Directoryf6e5d4c3b2a1f0e9/ (OpenMeteoSource)

      • metadata.json
      • data_20230101T000000_20240601T000000.parquet

Cache files are Parquet, so the UTC index, dtypes and floats round-trip exactly.

What is recorded as cached

A range counts as cached only where the source actually returned data, and never later than two days before today (UTC midnight). Data close to now may still be published or completed: actual load and generation appear about an hour after delivery (often later), next-day prices after the auction, weather forecasts after the model run finishes. The recent part is returned for the current run but fetched again next time, so a daily forecast run with cache=True always sees the newly published data.

A source can also say that some of its own rows are not settled yet. When the frame returned by fetch carries df.attrs["settled_until"] (a UTC timestamp), the rows from that instant on are returned for the current run but are not recorded as covered, however old they are. They are fetched again next time, and the settled rows before that instant are served from disk. This is for data that stays missing or provisional for much longer than two days, such as a period that the provider has not published yet. The pipeline reads and removes the attribute right after fetch, so it never appears on the output frame.

A gap in an otherwise regular response: one missing hour in a day of hourly data, is never recorded as covered: only the contiguous run(s) of timestamps actually present are cached, so the hole is fetched again on the next run instead of silently being treated as covered and then lost for good. A hole is fetched from the start of its local day (but not before the start of the run), and the cached rows of that day are replaced by what the refetch returns. If the source returns nothing for the refetch (ENTSO-E answers “No matching data” with an empty frame), the rows already cached stay in the result. The day is a local day of the pipeline’s timezone. Without one, an EntsoeSource uses the zone its week-ahead daily values are stamped at, which is its area’s own zone. A source that carries a one-day value forward to the start of its fetch, such as the ENTSO-E week-ahead daily load forecast, would otherwise stamp the same day twice.

A multi-column source joins its columns on an outer index, so the merged rows run as far as the longest column even when a shorter one stopped early: for example load_actual during a multi-day TSO outage, while price keeps publishing day-ahead rows. To stop that outage’s cells from being recorded as covered (and then never refetched once the source backfills them), the recorded range is additionally capped at one step past the last value of any column that is still being published, one whose last value is newer than 14 days before now. A column whose last value is older than that is treated as discontinued (e.g. a PsrType ENTSO-E no longer reports) and does not cap coverage, so it can’t force a refetch forever.

Each recorded range stores when it was retrieved (retrieved_at in metadata.json).

Caution

ENTSO-E sometimes revises data long after publication. The cache keeps the version it downloaded. Clear it to pick up revisions.

Several processes

Writes go to a temporary file that is then renamed into place, and each source’s cache is protected by a file lock (.lock in its directory). When several processes (for example SLURM array tasks) start together on a cold cache, the first one downloads each source while the others wait and then read its result: every task gets identical data and the API is called once.

Caution

The lock relies on flock. On network file systems that do not support it, a plain lock file is used instead. If a job is killed while holding it, delete the .lock file by hand.

Missing or damaged files

A cache file that was deleted or cannot be read counts as missing: its range is removed from metadata.json and downloaded again. A broken metadata.json makes the whole source cache count as empty.

Cache key

# These create different cache directories:
EntsoeSource(country_code="PL", api_key="...", type=["load"])
EntsoeSource(country_code="PL", api_key="...", type=["load", "price"])
EntsoeSource(country_code="DE_LU", api_key="...", type=["load"])

get_cache_config() must contain every parameter that changes the output. Sources also put a schema_version there and bump it when their parsing changes, and the package bumps its cache format version when the file layout changes, so stale files are never served. Old directories are then simply ignored, and you can delete them to reclaim disk space.

Single-file cache (cache="path")

The combined source output (before transformers) is stored in one file:

cache value	Stored as
"data.csv" (or any other file name)	CSV
"data.parquet"	Parquet
An existing directory, a path ending in /, or a new path without a file suffix	data_<hash>.parquet inside that directory, one file per configuration

$VAR, ${VAR} and ~ are expanded. A variable that is not set raises an error before anything is downloaded.

Next to the file, <file>.meta.json records a fingerprint of the sources (class, parameters without api_key, cache config, size and modification time of input files such as CsvSource.file_path), the requested start and end, the epftoolbox2 version and the cache format version. If any of these differ, the file is rebuilt. The CSV reader restores the exact floats, dtypes and column names, so a reload equals a fresh run.

It also records the unsettled boundary (two days before “now”, same as the source cache) at the moment the file was written. A later run with a matching fingerprint is only served from the file if either the whole requested range was already past that boundary when it was written (nothing about it could still change), or the boundary hasn’t moved since. Otherwise the file is rebuilt. Without this, a fixed (non-relative) date range whose end is in the future would cache missing/not-yet-published rows for the unsettled tail once and keep serving that same gap forever, even after the real data was published, because nothing about the requested dates or sources ever changes to invalidate the fingerprint.

If a source declared rows unsettled (settled_until, see above), the earliest such instant over all sources is stored as settled_until in the sidecar too. A file whose settled_until is earlier than the requested end is never reused and is rebuilt on every run, because what it holds was settled only up to that instant.

The single-file cache uses the same temporary-file-and-rename writes and a lock (<file>.lock), so parallel processes pointed at the same file download the data once.

Supported Sources

Source	Source cache
EntsoeSource	✅ Yes
OpenMeteoSource	✅ Yes
CalendarSource	✅ Yes (including school_holidays, see below)
CsvSource	❌ No (reads from file)
EpftoolboxBenchmarkSource	❌ No (keeps its own verified copy of the file instead)

CalendarSource.get_cache_config() includes every constructor parameter that affects its output, plus the resolved market time zone, the installed holidays package version (its holiday data changes between releases) and a schema_version. A change to any of them gives a different cache key, so stale values are never served. regimes is keyed by the periods it resolves to (name, start and end), so a change to the regime catalog gives a different key as well.

With school_holidays on, the cache key also holds the mode ("national" or "all"). Past school holidays are cached like any other data, so a rerun does not download them again. Dates after the end of the last school-holiday period that the OpenHolidays API has published are unknown and come out as NaN. CalendarSource declares them unsettled (settled_until, see What is recorded as cached), so they are never recorded and are requested again on every run until the API has published them. Then only those dates are fetched and the cached past is read from disk. A country for which the API has no school holidays at all is requested on every run for the same reason, because nothing about it is settled. The other sources of the pipeline are cached as usual.

Caution

The OpenHolidays API can also correct data it has already published, the same way ENTSO-E does. The cache keeps the version it downloaded. Clear it to pick up a correction to past dates. Only the dates that were still unpublished are refreshed on their own.

The single-file cache (cache="path/to/file.parquet") stores the earliest settled_until of its sources, so a file is rebuilt on every run while the API has not published the whole requested range. Once it has, the file is reused.

Clearing Cache

import shutil

# Clear all source caches
shutil.rmtree(".cache/sources")

# Or clear one source
shutil.rmtree(".cache/sources/a1b2c3d4e5f6a7b8")

Tip

The source cache is stored relative to your working directory. Use version control to ignore the .cache folder.