Skip to content

Datasets

fpm-rs uses explicit ordinary (version 1) and spectral (version 2) profiles with a local loader, regardless of where data came from:

dataset_spec + dataset files -> DatasetLoader -> Dataset -> reconstruction

The complete language-neutral contract is dataset_spec.md. Source-specific conversion remains outside this repository. This repository's registry distributes already-converted bundles that use the same local loader.

Open a local bundle

The standard entry point is dataset.json at the bundle root:

dataset-root/
  dataset.json
  measurements.json
  configuration.json
  frames/
  corrections/             # optional
  ground-truth.json         # optional
  valid-mask.json           # optional

Rust loading is explicit and offline:

use fpm_rs::{Result, datasets::DatasetLoader};

fn main() -> Result<()> {
    let dataset = DatasetLoader::new("/data/converted-fpm")?.load()?;
    let problem = dataset.reconstruction_problem()?;
    println!("{} frames", problem.measurements.frame_count());
    Ok(())
}

DatasetLoader validates safe paths, measurements, configuration and compiled models, optional truth and masks, provenance, and units. It never accesses the network.

Select frames and crop an ordinary dataset

Rust's Dataset::subset builder accepts FrameSelector::All, EveryNth(step), or ordered unique Indices, plus an optional detector Rect. Python resolves the same selection in one call:

subset = dataset.subset(frames=[2, 0], crop=(16, 32, 64, 64))
problem = subset.reconstruction_problem()
print(subset.spatial_crop, subset.measurements.shape)
# Alternatively select 0, 2, 4, ... and retain the full image:
frame_subset = dataset.subset(every_nth_frame=2)

Indices refer to the loaded acquisition, with the explicit supplied order preserved. frames and every_nth_frame are mutually exclusive. Crop order is (row, column, height, width) in original detector pixels, with positive sizes and containment checked before construction. Its object-space mapping must land on integer reconstruction pixels. Cropping owns new measurements and correction arrays, recompiles selected models, and crops optional truth/masks; it does not resample data. Python releases the GIL during subset construction.

DatasetSubset::spatial_crop / subset.spatial_crop returns the resolved rectangle, including the full image when no spatial crop was requested. Frame metadata preserves original acquisition indices and optional illumination identifiers separately from reindexed model sources. Dataset provenance and measurement units are retained. Use the subset-aware benchmark workflow to export these selections to JSON, CSV, and normalized Parquet tables. This is an in-memory selection API; the on-disk dataset contract is unchanged.

Open an explicit spectral bundle

Version 2 stores stable channel IDs, positive distinct vacuum wavelengths, response/weight provenance, per-channel compiled kernels on a common grid, and an explicit sparse detector exposure plan. Frames remain grayscale. External converters produce this metadata; RGB image planes are not inferred channels. The dataset specification defines the strict profile. Channel kernels can be emitted by ImagePlaneModel.save_json(path=...) from Python.

spectral = fpm.load_spectral_dataset(path="/data/converted-spectral-fpm")
problem = spectral.reconstruction_problem()
print(spectral.model.channel_ids, spectral.response_provenance)
result = fpm.SpectralAlternatingProjection(iterations=50).run(problem=problem)

In Rust use DatasetLoader::new(path)?.load_spectral()?, then reconstruction_problem(). Local loading stays offline and rejects missing response provenance, wavelength/kernel mismatches, incompatible common grids, invalid sparse references, unsafe paths (including escaping symlinks), and non-grayscale frames. Optional truth/masks remain channel ordered. Ordinary load and spectral load_spectral reject the other profile explicitly.

For a registered spectral ID, use registry.open_spectral(id) in either language, or fpm-datasets open ID --spectral. Registry profile versions 1 and 2 share verified downloads, staging, cache repair and cleanup; the registry schema itself remains version 1.

Discover and open registered datasets

The committed dataset_registry.json is an initially empty strict version-1 registry. By default installed clients read:

https://raw.githubusercontent.com/hgrecco/fpm-rs/main/dataset_registry.json

In Rust, use DatasetRegistry when an identifier should be downloaded on demand:

use fpm_rs::{Result, datasets::DatasetRegistry};

fn main() -> Result<()> {
    let registry = DatasetRegistry::from_defaults()?;
    for item in registry.list()? {
        println!("{} {} cached={}", item.entry.id, item.entry.version, item.cached);
    }
    let dataset = registry.open("example-led-array-dataset")?;
    dataset.reconstruction_problem()?;
    Ok(())
}

open reuses a valid current cache entry. Otherwise it downloads the immutable tar.zst archive, enforces its declared byte size, verifies SHA-256, extracts it into staging, validates it with DatasetLoader, and atomically installs it. The downloaded archive is discarded after installation.

Python exposes the same operations:

import fpm_rs as fpm

registry = fpm.DatasetRegistry()
for item in registry.list():
    print(item.id, item.version, item.cached)

dataset = registry.open("example-led-array-dataset")
problem = dataset.reconstruction_problem()

For a one-off default open, use datasets::open_dataset(id) in Rust or fpm_rs.open_dataset(id) in Python. Registry-backed operations may access the network; Python releases the GIL while they run.

Configuration and cache

Explicit constructor or CLI values take precedence over environment variables, which take precedence over defaults.

Setting Environment variable Default
Registry FPM_RS_DATASET_REGISTRY_URL Repository raw dataset_registry.json
Cache FPM_RS_DATASET_CACHE_DIR Platform cache directory under fpm-rs/datasets

Registry sources may be HTTP(S), file:// URLs, or filesystem paths. Successful registry responses are cached by source URL. Clients try the configured source first and use its matching snapshot only when retrieval fails.

The managed layout is:

<cache>/
  .fpm-rs-dataset-cache
  registries/<source-hash>.json
  datasets/<id>/<version>/
    dataset.json
    .fpm-rs-install.json
    ...
  partial/                       # transient only

Cache cleanup refuses unmarked roots and symbolic links. clean(id) removes all cached versions of one identifier; clean_all() removes the complete managed cache. A corrupt installed bundle is removed and downloaded once more when opened.

Command line

The Cargo package and Python wheel both install fpm-datasets:

fpm-datasets list
fpm-datasets download example-led-array-dataset
fpm-datasets download --all
fpm-datasets open example-led-array-dataset
fpm-datasets clean example-led-array-dataset
fpm-datasets clean --all

Every command accepts --registry-url URL and --cache-dir PATH before the subcommand. list reports ID, version, cache status, archive size, and title. open ensures the dataset is present, validates it, and prints its path and shapes.

Producing bundles

Conversion pipelines emit the files and metadata required by dataset_spec.md. Registry archives must contain dataset.json at archive root and must have immutable URLs, exact compressed sizes, and SHA-256 values. The registry carries discovery, license, citation, and source metadata; optical and frame-level metadata remain authoritative inside the bundle.