Ir para o conteúdo

Component: Research Experiments

Responsibility

Research experiments bring together the scientific specification, the manifest, the TrainingLocalModel code, dataset loaders, and experiment-execution applications.

Location

research/
  experiments/
    {experiment_id}/
      specification.md
      manifest.yaml
      docker.env.example
      src/main/java/

Current experiments

Experiment Purpose Execution Personal data (LGPD)
exp-har-smartphones-001 federated classification with UCI HAR Smartphones scripts/run-har-experiment.sh or scripts/run-har-experiment-docker.sh Personal data (Art. 5, I)
exp-ecg5000-anomaly-001 ECG anomaly detection with a federated MLP Autoencoder scripts/run-ecg-experiment.sh Sensitive health data (Art. 5, II)
exp-mimic-v31-icu-mortality-001 federated ICU mortality prediction with a tabular centroid-based baseline scripts/package-experiment.sh exp-mimic-v31-icu-mortality-001 to generate the package and dynamic deploy via /experiments/deploy Sensitive health data (Art. 5, II)
exp-mnist-digits-001 federated classification of handwritten digits (MLP 784→128→10) on MNIST dynamic registration via MnistSessionFactory//experiments/deploy No personal data (digit images)
exp-ergonomic-risk-001 federated binary classification of ergonomic risk scripts/package-experiment.sh exp-ergonomic-risk-001 and dynamic deployment Depends on the source records; assess before using identifiable data
exp-synthetic-smoke-001 not research — end-to-end platform validation on synthetic data with a known ground truth scripts/run-smoke-experiment.sh, or docker compose --profile smoke up No personal data (generated in process)

exp-ecg5000-anomaly-001 also has its own pom.xml in research/experiments/exp-ecg5000-anomaly-001/ (artifactId equal to experimentId), proving that the same source code compiles as a standalone Maven module — a prerequisite for packaging via sh scripts/package-experiment.sh exp-ecg5000-anomaly-001 (see docs/components/15-experiment-packaging.md). The legacy path embedded in federated-rest-api (via build-helper-maven-plugin) remains intact; the standalone module is additive and does not replace the current production path.

Inputs

  • raw dataset or institutional source;
  • scientific contract with readable consentTermsMarkdown (mandatory for personal data);
  • researcher's test dataset, when defined in the contract;
  • execution manifest;
  • TrainingLocalModel Java code;
  • list of participating clients.

Outputs

  • data partitioned into data/{client_id}/{experiment_id};
  • local updates (with differential privacy for ECG);
  • metrics;
  • execution reports;
  • auditable artifacts.
  • central evaluation metrics for local models and the global model when a contractual test dataset exists.

Contractual test dataset

The contract may declare a researcher-defined EvaluationDatasetSpec. This dataset is used by the researcher, orchestrator, aggregator, or an authorized evaluator to measure local models and the candidate global model.

Rules:

  • The test dataset is never sent to clients.
  • The test dataset must not be materialized under data/{client_id}/{experiment_id}.
  • The contract records datasetId, ownerId, uri, hash, metricName, and distributionPolicy.
  • distributionPolicy must be researcher_only_never_to_clients.
  • When evaluateClientModels=true, submitted local models are evaluated centrally after the round.
  • When evaluateGlobalModel=true, the candidate global model is evaluated centrally before any promotion.

Every experiment with identifiable personal data must carry the readable consent text in consentTermsMarkdown. HAR and ECG load it from <identity>/<consentContract> in their packaged research.xml; a missing or incomplete term prevents session construction. The web monitor displays the Markdown source to the participant/reviewer.

ECG5000 — official split and contractual evaluation

The exp-ecg5000-anomaly-001 experiment now uses the dataset's official split:

  • ECG5000_TRAIN is partitioned across the 5 federated clients with seed=42;
  • ECG5000_TEST remains the researcher's central holdout;
  • the holdout is described in the contract via EvaluationDatasetSpec;
  • the holdout is never materialized under data/{client_id}/{experiment_id}.

As a result, federated training avoids leakage between train and test, and AggregationReport now exposes f1_anomaly_detection_test on the official test set.

Official sampled replay for HAR/ECG e2e tests

The e2e tests with real datasets use a per-client cap to keep duration and MQTT volume reproducible, without altering the data-acquisition route:

Experiment Variable Value used in e2e Preflight minimum
HAR HAR_MAX_WINDOWS_PER_CLIENT 20 windows/subject 12
ECG5000 ECG_MAX_BEATS_PER_CLIENT 50 beats/client 50

HarMqttSimulationAdapter and Ecg5000MqttSimulationAdapter apply the cap to the prefix of each client's natural partition before publishing the in-memory repository over MQTT. The order is stable: HAR preserves the TRAIN split order by subject and ECG preserves the partition with seed=42. The JVM property takes precedence over the environment variable; a missing or invalid value publishes the entire partition.

The sampled replay does not write data directly to local storage, nor does it skip validations: the session still consumes MQTT via DataComponent.subscribeToMqttSource, materializes the snapshot, and runs the preflight normally. A cap below the contract minimum is not silently corrected and must fail the preflight; therefore, changing limits or the class distribution remains an explicit experimental decision.

Deployment and multi-round training suite — MNIST (BL-190)

modules/federated-system-tests/src/test/java/br/ufma/lsdi/research/mnist/MnistDeployAndTrainE2ETest.java builds the real federated session for exp-mnist-digits-001 (MnistFederatedSession, the same code used in production by MnistSessionFactory) and runs 5 rounds in sequence, verifying at each round: quorum satisfied, all clients participating, each client trained with samples acquired via MQTT, and the global model version advancing monotonically. It uses InMemoryMqttGateway (synchronous delivery, no real broker or Docker) — unlike the HAR/ECG e2e tests above (which require Docker plus an ephemeral Testcontainers broker), this suite runs in any environment and is tagged @Tag("integration"), not @Tag("e2e"), so it is part of sh scripts/run-tests.sh --fast.

Since the real MNIST dataset (~110MB) is not versioned in the repository, MnistMqttSimulationAdapter automatically falls back to a deterministic synthetic dataset (10 samples/client, one per class) when no real CSV is present — the suite still exercises the full production path (MQTT publication, DataComponent, preflight, FedAvg, local training via MnistTrainingLocalModel), only the accuracy stays close to chance (10 classes ≈ 0.1). For this reason, the suite's assertions validate structural progress per round (quorum, client count, model version advancement), not accuracy convergence.

For the suite to have access to the classes of exp-mnist-digits-001 (including package-private constructors used only in tests), research/experiments/exp-mnist-digits-001/src/main/java was added to the test sources of federated-system-tests via build-helper-maven-plugin (the same mechanism already used for DT/HAR/ECG5000), and the federated-experiment-packager dependency (used by CsvTabularDatasetSchema) was added to that module's pom.xml.

Differential Privacy in ECG (LGPD Art. 46)

The exp-ecg5000-anomaly-001 experiment processes health data (ECG), which qualifies as sensitive personal data (Art. 5, II). To mitigate membership inference attacks, Ecg5000TrainingLocalModel applies DP-SGD (Abadi et al., CCS 2016) to the weight vector before federated publication:

  1. L2 clip — bounds the vector's norm to dp_clip_norm (default: 1.0).
  2. Gaussian noise — adds N(0, σ²·C²·I) where σ = dp_noise_multiplier (calibrated: 0.5; reduced from legacy 1.1 to prevent catastrophic reconstruction loss degradation in advanced rounds while guaranteeing formal $(\varepsilon, \delta)$ privacy bounds with $\varepsilon \approx 3.2, \delta = 10^{-4}$).

DP remains enabled by default in the contract and the round hyperparameters. Since federated training preserves the official TRAIN/TEST split, formal (ε, δ) accounting is strictly calculated based on the effective size of TRAIN distributed among clients.

The DP hyperparameters are configurable via RoundConfig:

Hyperparameter Federated phase Fine-tune phase
dp_clip_norm 1.0 1.0
dp_noise_multiplier 0.5 0.3

The ECG contract's PrivacyPolicy reflects differentialPrivacyEnabled=true, noiseMultiplier=0.5.

See the Fabric component documentation and the federated client documentation for the current privacy and consent implementation.

Advanced Fine-Tuning and Optimization Strategies (BL-288)

To address heterogeneity, client drift, non-IID distribution, and severe class imbalances across edge devices, the experimental suite implements refined algorithmic architectures and server aggregation strategies:

1. UCI HAR (exp-har-smartphones-001)

  • Server Aggregation: FedAdam ($\eta_g = 1.0, \beta_1 = 0.9, \beta_2 = 0.99, \tau = 10^{-3}$) replacing vanilla FedAvg, mitigating client gradient drift.
  • Proximal Term Bounding: FedProx proximal parameter $\mu$ calibrated to $0.02$ (strictly bounded $\mu \le 0.03$), preventing the representation collapse observed when $\mu \ge 0.05$.
  • Regularization & Generalization: Dropout rate $0.10$, label smoothing $0.05$, local epochs $E=18$, initial learning rate $\eta_l = 0.0003$.
  • Stochastic Weight Averaging (SWA): Enabled from round 5 with decay $0.9$ to smooth local optima across heterogeneous mobile subjects.

2. ECG5000 Anomaly Detection (exp-ecg5000-anomaly-001)

  • Server Aggregation: FedAdam for stabilized autoencoder latent code consensus.
  • DP Noise Calibration: $\sigma = 0.5, C = 1.0$ (calibrated from 1.1 to avoid catastrophic divergence in later rounds).
  • Decision Threshold Optimization: Calibrated threshold $k=1.8$ (reconstruction error $> \mu_{\text{err}} + 1.8 \cdot \sigma_{\text{err}}$) maximizing Youden's J statistic ($J = \text{Sensitivity} + \text{Specificity} - 1$) and guaranteeing clinical recall $\ge 0.99$.
  • Ensemble Autoencoder Architecture: Multi-seed ensemble ($S=3$, seeds 42, 137, 271) with fine-tuning (50 epochs, $\eta_l = 0.0001$).

3. MIMIC v3.1 ICU Mortality (exp-mimic-v31-icu-mortality-001)

  • Imbalance Handling: Implementation of Focal Loss ($\alpha=0.25, \gamma=2.0$) and class-weighted penalty ($w_{\text{mort}}=7.0, w_{\text{surv}}=1.0$) for 10-15% minority mortality prevalence.
  • Robust Feature Preprocessing: Median and interquartile range (IQR) normalization per clinical signal to withstand extreme ICU sensor outliers.
  • Differential Privacy: $\sigma = 0.5, \varepsilon = 2.8$ for sensitive clinical biometric aggregates.

4. MNIST Digits (exp-mnist-digits-001)

  • Heterogeneous Partitioning: Non-IID Dirichlet distribution ($\alpha = 0.5$).
  • Client/Server Optimizers: Local AdamW with weight decay $10^{-4}$ and server-side FedAdam momentum.

5. Ergonomic Risk (exp-ergonomic-risk-001)

  • Asymmetric Cost Matrix: $w_{\text{risco}} = 3.0, w_{\text{seguro}} = 1.0$ to eliminate occupational false negatives.
  • Dynamic Thresholding: Youden's J calibrated cutoff on probabilistic outputs.

MIMIC v3.1 — federated ICU mortality

The exp-mimic-v31-icu-mortality-001 experiment introduces an auditable clinical baseline for the MIMIC v3.1 cohort, preserving the Java-only design and the framework's standard flow:

  • local tabular dataset materialized per client under data/{clientId}/exp-mimic-v31-icu-mortality-001/;
  • mandatory reading via DataComponent, with no direct access from training to the raw dataset;
  • MimicV31TrainingLocalModel training a federated binary centroid classifier (FEDERATED_CENTROID_CLASSIFIER) over 10 signals aggregated during the first 24h of admission;
  • MimicV31SessionFactory enabling packaging and dynamic deployment as ExperimentPackage v1;
  • synthetic fallback for development when cohort.csv is not present in the dataset root;
  • optional differential privacy applied to the local update when the round's PrivacyPolicy comes with differentialPrivacyEnabled=true.

Clinical features aggregated per stay:

  • age_years
  • heart_rate_mean
  • map_mean
  • spo2_mean
  • resp_rate_mean
  • temperature_c
  • wbc
  • creatinine
  • lactate
  • charlson_index

Target:

  • hospital_mortality_48h

Minimum expected structure in the raw dataset:

{datasetRoot}/cohort.csv

Expected columns:

record_id,client_id,age_years,heart_rate_mean,map_mean,spo2_mean,resp_rate_mean,
temperature_c,wbc,creatinine,lactate,charlson_index,hospital_mortality_48h

Packaging:

sh scripts/package-experiment.sh exp-mimic-v31-icu-mortality-001

The generated package references br.ufma.lsdi.research.mimic.MimicV31SessionFactory, preserving the same dynamic-deploy contract already used by the project's packageable experiments.

Docker organization

Experiments must not depend on fixed host paths. Data, artifact, runtime, and research paths must be resolved via ExperimentRuntimePaths.

Function Default local path Variable Path in container
Per-client data data HDT_AF_DATA_ROOT /app/data
Off-chain artifacts target/artifacts HDT_AF_ARTIFACT_ROOT /app/artifacts
Temporary runtime target/runtime* HDT_AF_RUNTIME_ROOT /app/runtime
Specifications and raw datasets research HDT_AF_RESEARCH_ROOT /app/research

For isolated client execution, the container must receive only the selected client's dataset:

data/{client_id}/{experiment_id} -> /app/data/{client_id}/{experiment_id}:ro

Full experiment applications, such as HAR, may mount data in write mode to materialize the datasets of all clients before the round.

Mandatory rules

  • All specification and experiment-specific code must live under research/.
  • Every experiment must have a manifest.yaml.
  • Every experiment must have a docker.env.example with the expected runtime variables.
  • The manifest must declare ml.algorithm_id, update_format, trainable_on_client, supports_federated_update, and supported_targets.
  • The manifest must declare the optimizer and the effective hyperparameters when the experiment has explicit fine-tuning.
  • Loaders may materialize data, but training must access data through DataComponent.
  • Loaders must separate federated training data from contractual test datasets.
  • Experiments must not reintroduce Python, gRPC, or dependencies outside the Java-only decision.
  • Experiments with health data must have differentialPrivacyEnabled=true in PrivacyPolicy and apply clip/noise compatible with the TrainingLocalModel.
  • Experiments with identifiable personal data must define readable consentTermsMarkdown in the contract.

Current state

The four experiments run end to end on the framework and write artifacts to target/artifacts. The HAR experiment is tuned with Adam, FedProx, dropout, L2, gradient clipping, and early stopping. The ECG experiment applies differential privacy (DP-SGD) to the federated updates. The MIMIC v3.1 experiment adds an auditable, packageable tabular baseline for ICU clinical scenarios with local data materialized via DataComponent.

Synthetic smoke experiment — end-to-end platform validation (BL-221)

exp-synthetic-smoke-001 is deliberately not a research experiment. It exists to answer one question the other experiments cannot: is the federated pipeline itself working?

When HAR or ECG fail to converge, the cause is ambiguous — platform defect, hyperparameters, or data. That ambiguity is what made the missing-client investigation slow. This experiment removes it: labels are a deterministic function of a known true weight vector over linearly separable data, so a correct pipeline must converge, and failure to converge is necessarily a platform defect.

Design decisions that matter

  • Logistic regression, not a neural net. Convex, so convergence on separable data is guaranteed; a failure accuses the pipeline, never the optimizer.
  • Strict quorum (all clients). HAR/ECG use max(3, ceil(n*0.6)), and that slack is what let a client go missing every round without failing anything. Here a missing client fails the test.
  • Differential privacy off. With noise around 7× the signal (as observed in ECG), the test could not distinguish correct aggregation from broken aggregation.
  • Data generated in process. Deterministic from (seed, clientId), so a Docker client container generates bit-for-bit the same partition the in-memory validation suite uses — no dataset to mount, download or version.

The validation suite

SmokePipelineValidationTest runs the same SmokeFederatedPipeline as the real app, swapping only the transport (InMemoryMqttGateway for Paho), and asserts all seven workflow steps. Two assertions carry most of the weight:

  • Aggregation math: the test independently recomputes the sample-weighted mean of the client updates and compares it position-by-position with the aggregator's global vector (1e-9). It verifies the arithmetic, not merely that a vector was produced.
  • Broadcast, by contradiction: if clients had not loaded round 1's global model, round 2 would restart from the seed and produce an identical vector. Differing vectors prove the broadcast landed.

The suite was verified against false positives by temporarily disabling step 7 in production code: exactly three tests failed (globalModelVersionAdvances, secondRoundStartsFromTheFirstRoundModel, globalLossDecreasesBetweenRounds) and no others.

Docker topology

docker compose --profile smoke up starts one aggregator container (DockerSmokeAggregatorDaemonApp) and four independent client containers (DockerSmokeClientDaemonApp). Clients are long-lived processes that listen for round announcements over MQTT and train on the finalization signal — the same external-participant flow as BL-210. What the Docker mode adds over the in-memory suite is real envelope serialization, signatures crossing process boundaries, and round deadlines that actually apply.

Reference result

Real run against mosquitto with mTLS, 2 rounds, 4 clients, seed 42:

Round Clients Quorum Avg local acc Global acc Global loss
1 4/4 yes 0.9583 0.9375 0.2929
2 4/4 yes 0.9833 0.9525 0.2246

Global accuracy went from 0.4050 (seed v0) to 0.9525, with no anomalies reported.