Ir para o conteúdo

Component: Experiment SDK

Responsibility

The Experiment SDK defines the Java contract that researchers must follow to implement local training algorithms. It separates the federated framework from the scientific code specific to each experiment, ensuring that no framework dependency leaks into the research code.

Implementation

Item Value
Module modules/federated-experiment-sdk
Java package br.ufma.lsdi.experiment.sdk
Main interfaces & exceptions TrainingLocalModel, TrainingContext, TrainingResult, TrainingException, ExperimentSession, ExperimentSessionFactory
ML catalog MlAlgorithmSpec, MlRuntimeTarget, MlAlgorithmCatalog
Utilities ExperimentRuntimePaths, ConsentTermsMarkdownReader, RoundEvaluationReport (EpochPoint lives in br.ufma.lsdi.common.model inside federated-common)

Main interface: TrainingLocalModel

This is the only interface the researcher needs to implement. The framework does not require any dependency beyond the SDK (and federated-common) to integrate a new experiment.

public interface TrainingLocalModel {

    /** Declares the algorithm and its supported targets. Validated before data access. */
    default MlAlgorithmSpec algorithmSpec() {
        return MlAlgorithmCatalog.CUSTOM_JAVA_TRAINING_JOB;
    }

    /**
     * Executes local training.
     *
     * @param context round metadata, authorized data view path, outputPath, hyperparameters
     */
    TrainingResult execute(TrainingContext context) throws TrainingException;
}

Mandatory rules for TrainingLocalModel:

  • Never open data/{client_id}/{experiment_id}/dataset.csv directly — read from context.authorizedDataView() (Path to authorized_view.csv).
  • Never write files outside context.outputPath().
  • Never connect to the network or external services.
  • Never return an empty vector or one containing NaN/Inf.
  • Declare algorithmSpec() with supported targets.
  • Produce per-epoch history (trainingHistory) and per-class accuracy (perClassAccuracy) when available — the web monitor displays them per client.

TrainingContext

Groups all execution context for a round without exposing the raw dataset.

Field Type Description
experimentId() String Experiment identifier
contractVersion() String Contract version
roundId() long Current round number
authorizedDataView() Path Path to the CSV generated by AuthorizedTrainingViewWriter — the only lawful data access
outputPath() Path Directory where local_update.bin, metrics.json, and data_profile.json are written
globalModelPath() Path Path to previous global model weights (nullable, used by FedProx/FedAdam)
hyperparameters() Map<String,String> Hyperparameters published in RoundConfig (epochs, lr, mu, dropout, …)
privacyPolicy() PrivacyPolicy Privacy policy metadata from RoundConfig
studyDesign() StudyDesign Study design metadata from the contract
aggregationPolicy() AggregationPolicy Aggregation policy metadata
dataWindow() TimeWindow Optional temporal window for the round

TrainingResult

Represents all artifacts and metrics produced by local training (also offers an 8-arg compatibility constructor without localMetric/localMetricName/trainingHistory/perClassAccuracy).

Field Type Description
localUpdatePath Path Path to local_update.bin (CSV of doubles)
metricsPath Path Path to metrics.json
dataProfilePath Path Path to data_profile.json
updateHash String SHA-256 of the contents of local_update.bin
updateNorm double L2 norm of the update vector
status TrainingStatus SUCCESS, FAILED, PREFLIGHT_FAILED, COMPLETED_NOT_SUBMITTED, UPDATE_SUBMITTED
sampleCount int Total number of samples used
positiveCount int Positive-class samples (binary classification)
localMetric double Value of the training metric (e.g., accuracy)
localMetricName String Metric name (e.g., "train_accuracy")
trainingHistory List<EpochPoint> Convergence history per epoch
perClassAccuracy Map<String,Double> Accuracy per class label (displayed as bars in the monitor)

EpochPoint (federated-common)

Record of one training epoch, propagated to the web monitor.

public record EpochPoint(int epoch, double loss, double accuracy, double gradientNorm) {}

The monitor displays a table of epochs and the per-class accuracy chart for each client on the round detail page. Essential for detecting client drift.


MlAlgorithmSpec

Declares the ML algorithm and the targets supported by the TrainingLocalModel.

public record MlAlgorithmSpec(
    String algorithmId,             // catalog ID (e.g., "federated_mlp_classifier")
    String algorithmFamily,         // family (e.g., "multiclass_classification")
    String updateFormat,            // update format (e.g., "neural_network_weights_csv")
    boolean trainableOnClient,      // whether it can be trained locally
    boolean supportsFederatedUpdate,// whether it produces a federatable update
    List<MlRuntimeTarget> supportedTargets  // compatible targets
)

MlRuntimeTarget

Enum with the supported execution targets:

Value Description
JVM_SERVER Standard JVM — server or desktop
EDGE_JVM Embedded JVM (limited resources)
ANDROID_JAVA Android with plain Java (lightweight algorithms)
ANDROID_LITERT Android LiteRT/TensorFlow Lite (trainable models)
ANDROID_ONNX Android ONNX Runtime (inference/validation)
MACOS_APPLE_SILICON_JAVA macOS Apple Silicon with AArch64 JDK
MACOS_APPLE_SILICON_DJL macOS Apple Silicon with DJL for execution/validation

The client validates ClientMlRuntimeSupport against the declared targets before granting access to the dataset.


MlAlgorithmCatalog

Static catalog of the framework's native algorithms:

ID Description
federated_mlp_classifier MLP for multiclass classification (HAR experiment)
federated_mlp_autoencoder MLP Autoencoder for anomaly detection (ECG experiment)
federated_centroid_classifier Centroid-based classifier for resource-constrained environments
federated_dt_regressor Regression for industrial digital twins (DT experiment)

Researchers can register algorithms outside the catalog by providing a custom algorithmId in MlAlgorithmSpec.


ExperimentSession and ExperimentSessionFactory (ADR-005)

Moved from br.ufma.lsdi.admin.AdminExperimentSession to br.ufma.lsdi.experiment.sdk so that packaged experiments (see docs/components/15-experiment-packaging.md) can implement them without depending on the entire federated-rest-api module.

public interface ExperimentSession {
    String experimentId();
    String contractVersion();
    FederatedExperimentRunResult executeRound(long roundId, String phase);
}

public interface ExperimentSessionFactory {
    ExperimentSession create(Path datasetRoot, String contractVersion, int maxClients);
}
  • ExperimentSession keeps infrastructure alive between rounds (loaded dataset, global weights, coordinator, aggregator) — each implementation (HAR, ECG, or a dynamically deployed experiment) initializes once and exposes executeRound for incremental execution.
  • ExperimentSessionFactory is the extension point referenced by the sessionFactoryClass field of a package's package-descriptor.json (ExperimentPackage) — validated and, for every round, actually loaded/executed inside a genuinely separate isolated OS process (own PID/JVM), never inside admin-api's own JVM (ExperimentDeploymentService/br.ufma.lsdi.admin.isolated.*, federated-rest-api module, BL-236).
  • datasetRoot may be null: the implementation resolves a default path (e.g., EcgSessionFactory delegates to Ecg5000DatasetLoader, which falls back to ExperimentRuntimePaths.datasetCacheRoot(...)).

ExperimentRuntimePaths

Resolves all system paths in a way that is portable between local execution and Docker.

Method Return Environment variable Default
dataRoot() Path HDT_AF_DATA_ROOT data
artifactRoot() Path HDT_AF_ARTIFACT_ROOT target/artifacts
runtimeRoot() Path HDT_AF_RUNTIME_ROOT target/runtime
researchRoot() Path HDT_AF_RESEARCH_ROOT research
deployedExperimentsRoot() Path HDT_AF_DEPLOYED_EXPERIMENTS_ROOT / DEPLOYED_EXPERIMENTS_ROOT target/deployed-experiments

deployedExperimentDirectory(experimentId, contractVersion, shortHash) resolves a directory isolated per deploy within deployedExperimentsRoot() — this prevents a new upload from overwriting a jar already in use by an isolated process spawned for a running session (see ADR-005, BL-236).

All experiments must use ExperimentRuntimePaths instead of literal strings.


Mandatory rules

  • Every TrainingLocalModel must declare algorithmSpec().
  • An algorithm with no target compatible with the client's environment must be blocked before data access.
  • An algorithm marked trainableOnClient=false must not be accepted for federated training.
  • Researcher code must live in research/experiments/{experiment_id}.
  • Dynamic deploy (ADR-005): ExperimentDeploymentService loads the package jar without verifying a signature, and the /experiments/deploy endpoint still has no access control of its own — a risk explicitly accepted by the operator of this MVP. Since BL-236, the loaded code always runs in a genuinely separate OS process, never inside admin-api's own JVM — see "Process isolation (BL-236)" and "Accepted risk" in docs/components/15-experiment-packaging.md. Jar signing and resource limits on the isolated process remain future hardening items, not implemented.
  • localUpdatePath must reference a file written under outputDir, not an arbitrary path.
  • updateHash must be the SHA-256 of the contents of local_update.bin; a mismatch with the hash recalculated by the client is grounds for rejection (ErrorCode.UPDATE_REJECTED).

Current state

The SDK supports JVM, Android Java, Android LiteRT, Android ONNX Runtime, macOS Apple Silicon Java, and macOS Apple Silicon DJL targets. The exp-har-smartphones-001 and exp-ecg5000-anomaly-001 experiments produce trainingHistory with EpochPoint and perClassAccuracy, visible in the web monitor per client and round.


Tests

Covered by federated-system-tests:

  • FederatedSystemTests — validates the TrainingLocalModel contract across all experiments end to end
  • ArchitectureComponentTestSuite — experiment SDK, TrainingLocalModel, TrainingResult, EpochPoint, supported algorithms

RoundEvaluationReport — evaluating a multi-round run

Renders a run of any ExperimentSession — per-client metrics per round, global-model evolution, and an explicit list of findings about the run's scientific quality. Pure logic, no I/O.

List<FederatedExperimentRunResult> rounds = ...;   // one per executeRound(...)
System.out.print(RoundEvaluationReport.render(rounds, requestedClients));

List<RoundEvaluationReport.Finding> findings =
        RoundEvaluationReport.analyse(rounds, requestedClients);

Why the findings exist

A MIMIC run of 5 rounds finished with every round in quorum, every client UPDATE_SUBMITTED, and train_accuracy of 1.000 — and still supported no conclusion: the metric was saturated, there was no holdout, and the global model's drift grew between rounds instead of shrinking. Every one of those observations required reading the numbers by hand.

A green pipeline is not a learning result. The report separates the two: what ran, and what the numbers permit you to claim.

Findings

Code Severity Detects
no-rounds ERROR nothing was executed
quorum-not-satisfied ERROR a round aggregated without the minimum clients
client-count-reduced WARNING fewer effective clients than requested — the dataset cannot support the requested federation, and the truncation is silent
not-converging WARNING drift between successive global models ends higher than it started
metric-saturated WARNING the local metric is identical across every client and round — no discriminating power
no-holdout-metric WARNING globalEvaluationMetric is NaN in every round — no central evaluation
update-exceeds-model WARNING mean ‖update‖ exceeds ‖global model‖ — each round rewrites rather than refines
no-promotion-decision INFO no round recorded PROMOTED/REJECTED

Convergence is judged by drift — L2 distance between successive global models. A converging run produces models that get closer together; if the final drift exceeds the first, the model is wandering. Needs at least 3 rounds to have an opinion.

Runners

Per-experiment main() apps render the report. MIMIC:

sh scripts/run-mimic-experiment.sh                      # synthetic fallback, 4 clients, 5 rounds
MIMIC_NUM_ROUNDS=10 sh scripts/run-mimic-experiment.sh /path/to/dataset
Variable Default Description
MIMIC_MAX_CLIENTS 4 clients per round (the dataset supports at most 4)
MIMIC_NUM_ROUNDS 5 federated rounds
MIMIC_DATASET_ROOT — dataset root; absent uses the synthetic fallback
MIMIC_USE_BROKER false true connects to a real MQTT broker via Paho

The app exits 1 when any ERROR-severity finding is present, so it can be chained in CI without parsing the report text.

Without a real cohort.csv, the run uses the synthetic fallback (10 admissions per client). That exercises the pipeline and says nothing about ICU mortality — the report flags it.