Component: Research Experiments¶
Responsibility¶
Research experiments bring together the scientific specification, the manifest, the TrainingLocalModel code, dataset loaders, and experiment-execution applications.
Location¶
research/
experiments/
{experiment_id}/
specification.md
manifest.yaml
docker.env.example
src/main/java/
Current experiments¶
| Experiment | Purpose | Execution | Personal data (LGPD) |
|---|---|---|---|
exp-har-smartphones-001 |
federated classification with UCI HAR Smartphones | scripts/run-har-experiment.sh or scripts/run-har-experiment-docker.sh |
Personal data (Art. 5, I) |
exp-ecg5000-anomaly-001 |
ECG anomaly detection with a federated MLP Autoencoder | scripts/run-ecg-experiment.sh |
Sensitive health data (Art. 5, II) |
exp-mimic-v31-icu-mortality-001 |
federated ICU mortality prediction with a tabular centroid-based baseline | scripts/package-experiment.sh exp-mimic-v31-icu-mortality-001 to generate the package and dynamic deploy via /experiments/deploy |
Sensitive health data (Art. 5, II) |
exp-mnist-digits-001 |
federated classification of handwritten digits (MLP 784→128→10) on MNIST | dynamic registration via MnistSessionFactory//experiments/deploy |
No personal data (digit images) |
exp-ergonomic-risk-001 |
federated binary classification of ergonomic risk | scripts/package-experiment.sh exp-ergonomic-risk-001 and dynamic deployment |
Depends on the source records; assess before using identifiable data |
exp-synthetic-smoke-001 |
not research — end-to-end platform validation on synthetic data with a known ground truth | scripts/run-smoke-experiment.sh, or docker compose --profile smoke up |
No personal data (generated in process) |
exp-ecg5000-anomaly-001 also has its own pom.xml in research/experiments/exp-ecg5000-anomaly-001/
(artifactId equal to experimentId), proving that the same source code compiles as a standalone
Maven module — a prerequisite for packaging via sh scripts/package-experiment.sh exp-ecg5000-anomaly-001
(see docs/components/15-experiment-packaging.md). The legacy path embedded in
federated-rest-api (via build-helper-maven-plugin) remains intact; the standalone module is
additive and does not replace the current production path.
Inputs¶
- raw dataset or institutional source;
- scientific contract with readable
consentTermsMarkdown(mandatory for personal data); - researcher's test dataset, when defined in the contract;
- execution manifest;
TrainingLocalModelJava code;- list of participating clients.
Outputs¶
- data partitioned into
data/{client_id}/{experiment_id}; - local updates (with differential privacy for ECG);
- metrics;
- execution reports;
- auditable artifacts.
- central evaluation metrics for local models and the global model when a contractual test dataset exists.
Contractual test dataset¶
The contract may declare a researcher-defined EvaluationDatasetSpec. This dataset is used by the researcher, orchestrator, aggregator, or an authorized evaluator to measure local models and the candidate global model.
Rules:
- The test dataset is never sent to clients.
- The test dataset must not be materialized under
data/{client_id}/{experiment_id}. - The contract records
datasetId,ownerId,uri,hash,metricName, anddistributionPolicy. distributionPolicymust beresearcher_only_never_to_clients.- When
evaluateClientModels=true, submitted local models are evaluated centrally after the round. - When
evaluateGlobalModel=true, the candidate global model is evaluated centrally before any promotion.
Consent terms (LGPD Art. 8 and 9)¶
Every experiment with identifiable personal data must carry the readable consent text in consentTermsMarkdown. HAR and ECG load it from <identity>/<consentContract> in their packaged research.xml; a missing or incomplete term prevents session construction. The web monitor displays the Markdown source to the participant/reviewer.
ECG5000 — official split and contractual evaluation¶
The exp-ecg5000-anomaly-001 experiment now uses the dataset's official split:
ECG5000_TRAINis partitioned across the 5 federated clients withseed=42;ECG5000_TESTremains the researcher's central holdout;- the holdout is described in the contract via
EvaluationDatasetSpec; - the holdout is never materialized under
data/{client_id}/{experiment_id}.
As a result, federated training avoids leakage between train and test, and AggregationReport now exposes f1_anomaly_detection_test on the official test set.
Official sampled replay for HAR/ECG e2e tests¶
The e2e tests with real datasets use a per-client cap to keep duration and MQTT volume reproducible, without altering the data-acquisition route:
| Experiment | Variable | Value used in e2e | Preflight minimum |
|---|---|---|---|
| HAR | HAR_MAX_WINDOWS_PER_CLIENT |
20 windows/subject | 12 |
| ECG5000 | ECG_MAX_BEATS_PER_CLIENT |
50 beats/client | 50 |
HarMqttSimulationAdapter and Ecg5000MqttSimulationAdapter apply the cap to the prefix of each
client's natural partition before publishing the in-memory repository over MQTT. The order
is stable: HAR preserves the TRAIN split order by subject and ECG preserves the partition with
seed=42. The JVM property takes precedence over the environment variable; a missing or
invalid value publishes the entire partition.
The sampled replay does not write data directly to local storage, nor does it skip validations:
the session still consumes MQTT via DataComponent.subscribeToMqttSource, materializes the snapshot, and
runs the preflight normally. A cap below the contract minimum is not silently corrected and
must fail the preflight; therefore, changing limits or the class distribution remains an
explicit experimental decision.
Deployment and multi-round training suite — MNIST (BL-190)¶
modules/federated-system-tests/src/test/java/br/ufma/lsdi/research/mnist/MnistDeployAndTrainE2ETest.java
builds the real federated session for exp-mnist-digits-001 (MnistFederatedSession, the same
code used in production by MnistSessionFactory) and runs 5 rounds in sequence,
verifying at each round: quorum satisfied, all clients participating, each client
trained with samples acquired via MQTT, and the global model version advancing
monotonically. It uses InMemoryMqttGateway (synchronous delivery, no real broker or Docker) —
unlike the HAR/ECG e2e tests above (which require Docker plus an ephemeral Testcontainers broker), this
suite runs in any environment and is tagged @Tag("integration"), not @Tag("e2e"), so it
is part of sh scripts/run-tests.sh --fast.
Since the real MNIST dataset (~110MB) is not versioned in the repository,
MnistMqttSimulationAdapter automatically falls back to a deterministic synthetic dataset (10
samples/client, one per class) when no real CSV is present — the suite still
exercises the full production path (MQTT publication, DataComponent, preflight, FedAvg,
local training via MnistTrainingLocalModel), only the accuracy stays close to chance (10
classes ≈ 0.1). For this reason, the suite's assertions validate structural progress per round
(quorum, client count, model version advancement), not accuracy convergence.
For the suite to have access to the classes of exp-mnist-digits-001 (including
package-private constructors used only in tests), research/experiments/exp-mnist-digits-001/src/main/java
was added to the test sources of federated-system-tests via build-helper-maven-plugin
(the same mechanism already used for DT/HAR/ECG5000), and the federated-experiment-packager
dependency (used by CsvTabularDatasetSchema) was added to that module's pom.xml.
Differential Privacy in ECG (LGPD Art. 46)¶
The exp-ecg5000-anomaly-001 experiment processes health data (ECG), which qualifies as sensitive personal data (Art. 5, II). To mitigate membership inference attacks, Ecg5000TrainingLocalModel applies DP-SGD (Abadi et al., CCS 2016) to the weight vector before federated publication:
- L2 clip — bounds the vector's norm to
dp_clip_norm(default: 1.0). - Gaussian noise — adds
N(0, σ²·C²·I)whereσ = dp_noise_multiplier(calibrated: 0.5; reduced from legacy 1.1 to prevent catastrophic reconstruction loss degradation in advanced rounds while guaranteeing formal $(\varepsilon, \delta)$ privacy bounds with $\varepsilon \approx 3.2, \delta = 10^{-4}$).
DP remains enabled by default in the contract and the round hyperparameters. Since federated training preserves the official TRAIN/TEST split, formal (ε, δ) accounting is strictly calculated based on the effective size of TRAIN distributed among clients.
The DP hyperparameters are configurable via RoundConfig:
| Hyperparameter | Federated phase | Fine-tune phase |
|---|---|---|
dp_clip_norm |
1.0 | 1.0 |
dp_noise_multiplier |
0.5 | 0.3 |
The ECG contract's PrivacyPolicy reflects differentialPrivacyEnabled=true, noiseMultiplier=0.5.
See the Fabric component documentation and the federated client documentation for the current privacy and consent implementation.
Advanced Fine-Tuning and Optimization Strategies (BL-288)¶
To address heterogeneity, client drift, non-IID distribution, and severe class imbalances across edge devices, the experimental suite implements refined algorithmic architectures and server aggregation strategies:
1. UCI HAR (exp-har-smartphones-001)¶
- Server Aggregation:
FedAdam($\eta_g = 1.0, \beta_1 = 0.9, \beta_2 = 0.99, \tau = 10^{-3}$) replacing vanilla FedAvg, mitigating client gradient drift. - Proximal Term Bounding: FedProx proximal parameter $\mu$ calibrated to $0.02$ (strictly bounded $\mu \le 0.03$), preventing the representation collapse observed when $\mu \ge 0.05$.
- Regularization & Generalization: Dropout rate $0.10$, label smoothing $0.05$, local epochs $E=18$, initial learning rate $\eta_l = 0.0003$.
- Stochastic Weight Averaging (SWA): Enabled from round 5 with decay $0.9$ to smooth local optima across heterogeneous mobile subjects.
2. ECG5000 Anomaly Detection (exp-ecg5000-anomaly-001)¶
- Server Aggregation:
FedAdamfor stabilized autoencoder latent code consensus. - DP Noise Calibration: $\sigma = 0.5, C = 1.0$ (calibrated from 1.1 to avoid catastrophic divergence in later rounds).
- Decision Threshold Optimization: Calibrated threshold $k=1.8$ (reconstruction error $> \mu_{\text{err}} + 1.8 \cdot \sigma_{\text{err}}$) maximizing Youden's J statistic ($J = \text{Sensitivity} + \text{Specificity} - 1$) and guaranteeing clinical recall $\ge 0.99$.
- Ensemble Autoencoder Architecture: Multi-seed ensemble ($S=3$, seeds 42, 137, 271) with fine-tuning (50 epochs, $\eta_l = 0.0001$).
3. MIMIC v3.1 ICU Mortality (exp-mimic-v31-icu-mortality-001)¶
- Imbalance Handling: Implementation of Focal Loss ($\alpha=0.25, \gamma=2.0$) and class-weighted penalty ($w_{\text{mort}}=7.0, w_{\text{surv}}=1.0$) for 10-15% minority mortality prevalence.
- Robust Feature Preprocessing: Median and interquartile range (IQR) normalization per clinical signal to withstand extreme ICU sensor outliers.
- Differential Privacy: $\sigma = 0.5, \varepsilon = 2.8$ for sensitive clinical biometric aggregates.
4. MNIST Digits (exp-mnist-digits-001)¶
- Heterogeneous Partitioning: Non-IID Dirichlet distribution ($\alpha = 0.5$).
- Client/Server Optimizers: Local AdamW with weight decay $10^{-4}$ and server-side FedAdam momentum.
5. Ergonomic Risk (exp-ergonomic-risk-001)¶
- Asymmetric Cost Matrix: $w_{\text{risco}} = 3.0, w_{\text{seguro}} = 1.0$ to eliminate occupational false negatives.
- Dynamic Thresholding: Youden's J calibrated cutoff on probabilistic outputs.
MIMIC v3.1 — federated ICU mortality¶
The exp-mimic-v31-icu-mortality-001 experiment introduces an auditable clinical baseline for the
MIMIC v3.1 cohort, preserving the Java-only design and the framework's standard flow:
- local tabular dataset materialized per client under
data/{clientId}/exp-mimic-v31-icu-mortality-001/; - mandatory reading via
DataComponent, with no direct access from training to the raw dataset; MimicV31TrainingLocalModeltraining a federated binary centroid classifier (FEDERATED_CENTROID_CLASSIFIER) over 10 signals aggregated during the first 24h of admission;MimicV31SessionFactoryenabling packaging and dynamic deployment asExperimentPackage v1;- synthetic fallback for development when
cohort.csvis not present in the dataset root; - optional differential privacy applied to the local update when the round's
PrivacyPolicycomes withdifferentialPrivacyEnabled=true.
Clinical features aggregated per stay:
age_yearsheart_rate_meanmap_meanspo2_meanresp_rate_meantemperature_cwbccreatininelactatecharlson_index
Target:
hospital_mortality_48h
Minimum expected structure in the raw dataset:
Expected columns:
record_id,client_id,age_years,heart_rate_mean,map_mean,spo2_mean,resp_rate_mean,
temperature_c,wbc,creatinine,lactate,charlson_index,hospital_mortality_48h
Packaging:
The generated package references br.ufma.lsdi.research.mimic.MimicV31SessionFactory, preserving the
same dynamic-deploy contract already used by the project's packageable experiments.
Docker organization¶
Experiments must not depend on fixed host paths. Data, artifact, runtime, and research paths must be resolved via ExperimentRuntimePaths.
| Function | Default local path | Variable | Path in container |
|---|---|---|---|
| Per-client data | data |
HDT_AF_DATA_ROOT |
/app/data |
| Off-chain artifacts | target/artifacts |
HDT_AF_ARTIFACT_ROOT |
/app/artifacts |
| Temporary runtime | target/runtime* |
HDT_AF_RUNTIME_ROOT |
/app/runtime |
| Specifications and raw datasets | research |
HDT_AF_RESEARCH_ROOT |
/app/research |
For isolated client execution, the container must receive only the selected client's dataset:
Full experiment applications, such as HAR, may mount data in write mode to materialize the datasets of all clients before the round.
Mandatory rules¶
- All specification and experiment-specific code must live under
research/. - Every experiment must have a
manifest.yaml. - Every experiment must have a
docker.env.examplewith the expected runtime variables. - The manifest must declare
ml.algorithm_id,update_format,trainable_on_client,supports_federated_update, andsupported_targets. - The manifest must declare the optimizer and the effective hyperparameters when the experiment has explicit fine-tuning.
- Loaders may materialize data, but training must access data through
DataComponent. - Loaders must separate federated training data from contractual test datasets.
- Experiments must not reintroduce Python, gRPC, or dependencies outside the Java-only decision.
- Experiments with health data must have
differentialPrivacyEnabled=trueinPrivacyPolicyand apply clip/noise compatible with theTrainingLocalModel. - Experiments with identifiable personal data must define readable
consentTermsMarkdownin the contract.
Current state¶
The four experiments run end to end on the framework and write artifacts to
target/artifacts. The HAR experiment is tuned with Adam, FedProx, dropout, L2, gradient
clipping, and early stopping. The ECG experiment applies differential privacy (DP-SGD) to the
federated updates. The MIMIC v3.1 experiment adds an auditable, packageable tabular baseline for
ICU clinical scenarios with local data materialized via DataComponent.
Synthetic smoke experiment — end-to-end platform validation (BL-221)¶
exp-synthetic-smoke-001 is deliberately not a research experiment. It exists to answer one
question the other experiments cannot: is the federated pipeline itself working?
When HAR or ECG fail to converge, the cause is ambiguous — platform defect, hyperparameters, or data. That ambiguity is what made the missing-client investigation slow. This experiment removes it: labels are a deterministic function of a known true weight vector over linearly separable data, so a correct pipeline must converge, and failure to converge is necessarily a platform defect.
Design decisions that matter¶
- Logistic regression, not a neural net. Convex, so convergence on separable data is guaranteed; a failure accuses the pipeline, never the optimizer.
- Strict quorum (all clients). HAR/ECG use
max(3, ceil(n*0.6)), and that slack is what let a client go missing every round without failing anything. Here a missing client fails the test. - Differential privacy off. With noise around 7× the signal (as observed in ECG), the test could not distinguish correct aggregation from broken aggregation.
- Data generated in process. Deterministic from
(seed, clientId), so a Docker client container generates bit-for-bit the same partition the in-memory validation suite uses — no dataset to mount, download or version.
The validation suite¶
SmokePipelineValidationTest runs the same SmokeFederatedPipeline as the real app, swapping
only the transport (InMemoryMqttGateway for Paho), and asserts all seven workflow steps. Two
assertions carry most of the weight:
- Aggregation math: the test independently recomputes the sample-weighted mean of the client
updates and compares it position-by-position with the aggregator's global vector (
1e-9). It verifies the arithmetic, not merely that a vector was produced. - Broadcast, by contradiction: if clients had not loaded round 1's global model, round 2 would restart from the seed and produce an identical vector. Differing vectors prove the broadcast landed.
The suite was verified against false positives by temporarily disabling step 7 in production code:
exactly three tests failed (globalModelVersionAdvances,
secondRoundStartsFromTheFirstRoundModel, globalLossDecreasesBetweenRounds) and no others.
Docker topology¶
docker compose --profile smoke up starts one aggregator container
(DockerSmokeAggregatorDaemonApp) and four independent client containers
(DockerSmokeClientDaemonApp). Clients are long-lived processes that listen for round
announcements over MQTT and train on the finalization signal — the same external-participant flow as
BL-210. What the Docker mode adds over the in-memory suite is real envelope serialization,
signatures crossing process boundaries, and round deadlines that actually apply.
Reference result¶
Real run against mosquitto with mTLS, 2 rounds, 4 clients, seed 42:
| Round | Clients | Quorum | Avg local acc | Global acc | Global loss |
|---|---|---|---|---|---|
| 1 | 4/4 | yes | 0.9583 | 0.9375 | 0.2929 |
| 2 | 4/4 | yes | 0.9833 | 0.9525 | 0.2246 |
Global accuracy went from 0.4050 (seed v0) to 0.9525, with no anomalies reported.