Ir para o conteúdo

Complete Tutorial — Create, Configure, Deploy, Simulate, and Monitor an Experiment (HDT-AF)

This tutorial consolidates into a single end-to-end flow what is currently spread across docs/experiments-guide.md, docs/execution-guide.md, docs/simulation-deployment-guide.md, and the components documented in docs/components/. It covers the five stages of a federated experiment's life cycle in the HDT-AF framework:

1. Experiment creation      → structure, manifest.yaml, TrainingLocalModel, contract
2. Data configuration       → data/{client_id}/{experiment_id}, DataComponent
3. Deployment                → already-registered path or packaging + dynamic upload
4. Simulation execution      → round-based session in the monitor, CLI scripts, or MQTT replay
5. Monitoring                → dashboard, rounds, metrics, tasks, ledger

For quick, always up-to-date reference, the source documents remain valid and are referenced at the end of each section. This tutorial does not replace them — it unites them into a single narrative, verified against the current state of the repository.


Prerequisites

Item Minimum Note
Java (JDK) 25 Required by the current root Maven reactor. The Android Gradle app has separate requirements.
Maven 3.8+ Required to build standalone modules and package experiments
MQTT Broker reachable at MQTT_BROKER_URL Default: tcp://localhost:1883
Docker 20+ Only if running clients in containers
lsof any Used by the monitor to automatically release port 8088

Quick check:

java -version
mvn -version
echo "${MQTT_BROKER_URL:-tcp://localhost:1883}"

Build the entire project before any step below:

sh scripts/compile.sh

This prepares target/classes and target/lib, used by all the run-* scripts.


Experiments currently in the repository

Before creating a new experiment, check whether it already exists. These are the experiments present under research/experiments/ and/or registered by default in the monitor:

experimentId Registered by default in the monitor Has its own folder in research/experiments/ Packageable (package-experiment.sh)
exp-dt-federated-001 Yes (bootstrap/demo) No standalone experiment source directory in this checkout No
exp-har-smartphones-001 Yes (bootstrap) — UCI HAR Smartphones Yes No (still uses the legacy hardcoded path)
exp-ecg5000-anomaly-001 Yes (bootstrap) — ECG5000, MLP Autoencoder + DP Yes Yes
exp-mimic-v31-icu-mortality-001 No — requires dynamic deployment Yes Yes
exp-mnist-digits-001 No — requires dynamic deployment Yes Yes
exp-ergonomic-risk-001 No — requires dynamic deployment Yes Yes
exp-synthetic-smoke-001 No — standalone smoke runner Yes No — use scripts/run-smoke-experiment.sh

The scripts/package-experiment.sh script currently recognizes four IDs: ECG5000, MIMIC v3.1, MNIST, and ergonomic risk. Other identifiers require their package configuration and session factory to be added to the script.


Part 1 — Creating the experiment

1.1 Where an experiment lives

research/experiments/{experiment_id}/
├── specification.md        ← hypothesis, dataset, model, federated protocol (scientific document)
├── manifest.yaml            ← executable technical metadata (mandatory)
├── docker.env.example       ← variables expected in the container
├── pom.xml                  ← mandatory only if the experiment is packageable/dynamically deployed
└── src/main/java/br/ufma/lsdi/research/{dominio}/
    ├── {Nome}TrainingLocalModel.java   ← local training algorithm (the only mandatory piece)
    ├── {Nome}FederatedExperimentApp.java ← orchestrates the full run via CLI
    ├── {Nome}DatasetLoader.java        ← partitions and materializes per-client data
    └── {Nome}SessionFactory.java       ← implements ExperimentSessionFactory (only for dynamic deployment)

1.2 manifest.yaml — mandatory metadata

experiment_id: meu-experimento-001
version: "1.0.0"
coordinator_id: researcher-001

metadata:
  title: "Readable title of the experiment"
  description: >
    Description of the problem, dataset, and model architecture.
  task_type: multiclass_classification   # or anomaly_detection, binary_classification, etc.
  dataset: NomeDoDataset
  synthetic_data: false

training:
  entrypoint: br.ufma.lsdi.research.meu.MeuTrainingLocalModel
  algorithm: federated_mlp_classifier      # must exist in MlAlgorithmCatalog
  update_format: neural_network_weights_csv
  runtime: pure_java_sdk
  hyperparameters:
    epochs: 20
    batch_size: 32
    learning_rate: 0.001

federation:
  num_clients: 5
  num_rounds: 10
  quorum_min_clients: 3
  aggregation_strategy: fedavg
  partition_strategy: random_shuffle_seed42

inputs:
  data_properties:
    - { name: "feature_1..feature_n", type: float, role: feature }
    - { name: label, type: integer, role: target }
  prohibited: [identificadores_diretos, dados_brutos_originais]

outputs:
  files: [local_update.bin, metrics.json, data_profile.json]
  metrics_allowed: [train_accuracy, sample_count, update_norm]

Mandatory manifest rules (see docs/components/13-research-experiments.md):

  • must declare ml.algorithm_id/algorithm, update_format, trainable_on_client, supports_federated_update (when applicable), and the supported targets;
  • experiments with health data or identifiable personal data must indicate this clearly, since the contract will require differentialPrivacyEnabled=true and/or readable consentTermsMarkdown.

1.3 Implementing TrainingLocalModel

This is the only piece of Java code the researcher needs to write to integrate an algorithm with the framework:

package br.ufma.lsdi.research.meu;

public class MeuTrainingLocalModel implements TrainingLocalModel {

    @Override
    public MlAlgorithmSpec algorithmSpec() {
        return MlAlgorithmCatalog.FEDERATED_MLP_CLASSIFIER;
    }

    @Override
    public TrainingResult execute(TrainingContext context, Path outputDir) {
        // 1. Load authorized data — NEVER open data/ directly
        List<double[]> features = context.authorizedDataView().features();
        int[] labels = context.authorizedDataView().labels();

        // 2. Train locally
        double[] weights = treinar(features, labels, context.hyperparameters());

        // 3. Serialize the update
        Path updatePath = outputDir.resolve("local_update.bin");
        Files.writeString(updatePath, VectorCodec.toCsv(weights), StandardCharsets.UTF_8);

        // 4. Write metrics.json and data_profile.json to outputDir

        // 5. Return TrainingResult with real metrics (never empty or NaN/Inf)
        return new TrainingResult(updatePath, metricsPath, profilePath,
                Hashing.sha256(...), VectorCodec.norm(weights), TrainingStatus.SUCCESS,
                features.size(), positiveCount, accuracy, "train_accuracy",
                epochPoints, perClassAccuracy);
    }
}

Strict rules — what never to do inside a TrainingLocalModel:

  • do not open data/{client_id}/{experiment_id}/dataset.csv directly — always use context.authorizedDataView();
  • do not write files outside outputDir;
  • do not open network connections or external services;
  • do not use Python, gRPC, or any library outside the Java ecosystem (the project's Java-only architecture rule);
  • do not return an update vector that is empty or contains NaN or Inf.

1.4 Defining the experiment contract (ExperimentContract)

The contract is assembled programmatically in the experimental application. Main blocks:

ExperimentContract contract = new ExperimentContract(
    "meu-experimento-001", "1.0.0", "researcher-001",
    "Objetivo do experimento", "tipo_de_gemeo_digital", "multiclass_classification",
    ContractStatus.ACTIVE, Instant.now(), Instant.now().plusSeconds(86400),
    Hashing.sha256(evidence), codeRegistry, dataSchema, studyDesign,
    aggregationPolicy, privacyPolicy, outputPolicy);
  • AggregationPolicy defines the strategy (fedavg), minimum client quorum, minimum per-client weight, and maxUpdateNorm (rejects outliers).
  • PrivacyPolicy defines whether there is differential privacy (epsilon, delta) and the minimum number of clients for aggregation (k-anonymity protection).
  • OutputPolicy defines which files and metrics the client can send, and the maximum payload size.
  • Experiments with health/personal data must set differentialPrivacyEnabled=true and consentTermsMarkdown (see Fabric and client privacy documentation).

1.5 Creating the experimental application ({Nome}FederatedExperimentApp)

Orchestrates the full cycle: loads/partitions the dataset → registers the contract → registers consents → seeds the global model v0 (seedGlobalModel) → runs the rounds (orchestrator.startRound → each client's training → aggregator.aggregate) → registers completion. Follow the pattern already established in research/experiments/exp-ecg5000-anomaly-001/src/main/java/.

1.6 Making the experiment packageable (optional, required for dynamic deployment)

For the experiment to be deployable to the monitor via upload (without editing/rebuilding federated-rest-api):

  1. Create its own research/experiments/{id}/pom.xml, with artifactId matching the experimentId.
  2. Implement {Nome}SessionFactory implements ExperimentSessionFactory in the same package as the session.
  3. Register the experiment in scripts/package-experiment.sh, adding a case with SESSION_FACTORY_CLASS and CONTRACT_VERSION (follow the pattern of exp-mimic-v31-icu-mortality-001 and exp-mnist-digits-001).

If this step is skipped, packaging fails with an explicit message asking for this registration.

References: docs/experiments-guide.md, docs/components/13-research-experiments.md, docs/components/09-experiment-sdk.md, docs/components/10-training-local-model.md.


Part 2 — Data configuration

2.1 Per-client data structure

data/
  {client_id}/
    {experiment_id}/
      dataset.csv     ← features + label, one sample per line
      metadata.json   ← origin, record count, notes

Mandatory metadata.json:

{
  "client_id": "3",
  "experiment_id": "meu-experimento-001",
  "record_count": 1200,
  "source": "Data origin / partitioning identification",
  "schema_version": "1.0.0",
  "notes": "How the data was partitioned for this client."
}

2.2 Who materializes the data

The experiment's own DatasetLoader is responsible for: reading the raw source (CSV, database, institutional file), partitioning by client, and writing dataset.csv + metadata.json under data/{client_id}/{experiment_id}/.

Runtime access during training is never direct file access: the DataComponent (module federated-local-cdm) encapsulates the read and delivers to FederatedClientService an authorizedDataView() derived from the dataset and the contract's policy. TrainingLocalModel consumes only that authorized view.

2.3 Datasets for each existing experiment

Experiment Dataset expected locally Automatic download
HAR (exp-har-smartphones-001) research/datasets/uci-har/UCI HAR Dataset (features.txt, activity_labels.txt, train/X_train.txt, train/y_train.txt, train/subject_train.txt, test/...) Yes, via scripts/run-har-experiment.sh with no argument
ECG5000 (exp-ecg5000-anomaly-001) research/datasets/ecg5000/ECG5000 (ECG5000_TRAIN.arff/.txt, ECG5000_TEST.arff/.txt) Yes, via scripts/run-ecg-experiment.sh with no argument
MIMIC v3.1 (exp-mimic-v31-icu-mortality-001) {datasetRoot}/cohort.csv with columns record_id,client_id,age_years,heart_rate_mean,map_mean,spo2_mean,resp_rate_mean,temperature_c,wbc,creatinine,lactate,charlson_index,hospital_mortality_48h No — synthetic fallback for development if cohort.csv does not exist
MNIST (exp-mnist-digits-001) Flattened CSV with 784 pixel columns (pixel_0..pixel_783) + label, pointed to via EXP_MNIST_DIGITS_001_DATASET (see §2.4) No

2.4 Environment variables for data location

Variable Default Description
HDT_AF_DATA_ROOT data Root of per-client partitioned data
HDT_AF_ARTIFACT_ROOT target/artifacts Root of off-chain artifacts
HDT_AF_RUNTIME_ROOT target/runtime* Root of temporary runtime files
HDT_AF_RESEARCH_ROOT research Root of research datasets and specifications
HAR_MAX_CLIENTS 30 Limits HAR experiment participants (useful for quick tests)
{EXPERIMENT_ID}_DATASET (e.g., EXP_MNIST_DIGITS_001_DATASET) — (datasetRoot=null) For experiments deployed dynamically via /experiments/deploy (MNIST, MIMIC): path to the real CSV. Key derived from experimentId in uppercase (AdminExperimentExecutor.datasetEnvKeyFor) — without it, the session falls back to ExperimentRuntimePaths.datasetCacheRoot(...)/synthetic data

2.5 Contractual test dataset (researcher holdout)

Several experiments (e.g., ECG5000, HAR) declare an EvaluationDatasetSpec in the contract: an official test split that is never sent to the clients nor materialized under data/{client_id}/{experiment_id}. It is used only by the researcher/Aggregator to evaluate local models and the candidate global model (distributionPolicy=researcher_only_never_to_clients). When creating a new experiment, clearly separate the federated training dataset (per client) from the central evaluation dataset.

References: docs/experiments-guide.md (Part 3), docs/components/13-research-experiments.md, docs/components/09-experiment-sdk.md (DataComponent).


Part 3 — Deploying the experiment

There are two valid paths, depending on whether the experiment is already known to the monitor.

3.1 Path A — experiment already registered in the monitor

Applies to exp-dt-federated-001, exp-har-smartphones-001, and exp-ecg5000-anomaly-001 (registered automatically by AdminApplicationBootstrap). Just start the monitor and use it — no need to package anything.

sh scripts/run-admin-web.sh

Go to http://127.0.0.1:8088, find the experiment on the dashboard, and open the details page.

3.2 Path B — dynamic deployment via upload (ExperimentPackage v1)

Required for exp-mimic-v31-icu-mortality-001, exp-mnist-digits-001, or any new experiment you have just registered in scripts/package-experiment.sh.

Step 1 — generate the package:

sh scripts/package-experiment.sh {experimentId}
# → target/packages/{experimentId}.hdtpkg.zip

Package and submit to the monitor in the same command:

sh scripts/package-experiment.sh {experimentId} --submit
sh scripts/package-experiment.sh {experimentId} --submit --client-ids 1,2,3
sh scripts/package-experiment.sh {experimentId} --submit --platform-url http://127.0.0.1:8088

With --submit, the script sends the zip via multipart/form-data to POST /experiments/deploy and writes the HTTP response to target/packages/{experimentId}.deploy.response.

Step 2 — start the monitor (if not already running):

sh scripts/run-admin-web.sh

Step 3 — upload the package through the browser (if you did not use --submit):

Go to http://127.0.0.1:8088/experiments/deploy, select the .hdtpkg.zip file, enter clientIds as CSV (optional), and submit the form. On success, the monitor redirects (303) to /experiments/{id}?version={contractVersion}.

Step 4 — validate the deployment: confirm that the experiment appears on the dashboard, that the details page opens normally, that the contract version is correct, and that it is ready for execution/simulation.

3.3 What happens internally during dynamic deployment

  1. Extracts the zip with checks for zip slip, zip bomb, SHA256SUMS, and packageFormatVersion.
  2. Moves the content to a directory versioned by contractVersion + shortHash (redeploying the same content is idempotent; new content uses a new directory).
  3. Validates sessionFactoryClass by launching a disposable isolated OS process (java subprocess, own PID/JVM) and sending it a VALIDATE command — the class is loaded and instantiated in that separate process, never inside admin-api's own JVM (BL-236).
  4. Registers an IsolatedExperimentSessionFactory (not the loaded factory itself) with priority over the legacy hardcoded switch (HAR/ECG/DT continue working unchanged); every later round also runs in an isolated process, launched on demand.
  5. Creates the ExperimentContract already in ACTIVE status and registers it in AdminMonitoringService.

3.4 Security notice

The /experiments/deploy endpoint itself currently has no authentication in the MVP — anyone with access to port 8088 can upload a package. This is a risk the project team has explicitly accepted for the endpoint's access control. What changed (BL-236): the uploaded code no longer runs inside admin-api's own process — it always executes in a genuinely separate OS process, so it can no longer read/corrupt the monitor's own memory, credentials, or other experiments' state. Resource limits (CPU/memory/time) on that isolated process are not yet implemented. Use this flow only on a trusted network, never expose port 8088 publicly.

References: docs/simulation-deployment-guide.md, docs/components/15-experiment-packaging.md, docs/components/01-admin-api-monitor-web.md.


Part 4 — Running the simulation

There are three ways to run the simulation; use whichever best fits your workflow.

With the monitor running and the experiment deployed (Part 3):

  1. Open the experiment and click the ▶ Session tab.
  2. Fill in the fields:
  3. type: har or ecg (replay adapters currently available)
  4. path: dataset path, e.g. research/datasets/uci-har/UCI HAR Dataset or research/datasets/ecg5000/ECG5000
  5. maxClients: maximum number of simulated clients
  6. totalRounds: total number of rounds
  7. Click Start session (POST /sim-session/start).
  8. Advance with Send round (one round at a time) or ⏩ Run All Remaining Rounds (run-all, which also stops automatically and marks ERROR if any round fails).
  9. Use Pause/Resume as needed; Stop clears the session.

Possible session states: ACTIVE, PAUSED, DONE, ERROR. In ERROR, the current round does not advance automatically — check last_message for the diagnosis.

Programmatic state query:

curl http://127.0.0.1:8088/api/experiments/{id}/sim-session

4.2 Execution via command-line scripts (without the monitor)

For the three experiments with a dedicated script:

# DT — industrial digital twins
The DT demo is launched from the monitor simulation session. The old standalone
`run-simulator.sh` and `run-simulator-docker.sh` wrappers are not present in this checkout.

# HAR
sh scripts/run-har-experiment.sh
sh scripts/run-har-experiment.sh "research/datasets/uci-har/UCI HAR Dataset"
HAR_MAX_CLIENTS=5 sh scripts/run-har-experiment.sh "research/datasets/uci-har/UCI HAR Dataset"
sh scripts/run-har-experiment-docker.sh "research/datasets/uci-har/UCI HAR Dataset"

# ECG5000
sh scripts/run-ecg-experiment.sh
sh scripts/run-ecg-experiment.sh "research/datasets/ecg5000/ECG5000"

To follow it in real time, run the monitor in one terminal and the experiment in another — both both use the required Fabric ledger; configure the same Fabric network, channel, chaincode, and MSP identity for each process:

# Terminal 1
sh scripts/run-admin-web.sh
# Terminal 2
sh scripts/run-har-experiment.sh "research/datasets/uci-har/UCI HAR Dataset"

4.3 MQTT replay (simulate IoT devices publishing data)

Useful for testing the data ingestion path as if it came from real sensors, feeding the DataComponent:

sh scripts/run-iot-simulator.sh har
sh scripts/run-iot-simulator.sh har "research/datasets/uci-har/UCI HAR Dataset" 5   # 5 clients
sh scripts/run-iot-simulator.sh ecg
sh scripts/run-iot-simulator.sh ecg "research/datasets/ecg5000/ECG5000" 3
# Replay currently supports the har and ecg adapters.

The simulator publishes to iot/clients/{clientId}/entities/{entityId}/observations using the broker configured in MQTT_BROKER_URL. Each round replays a deterministic slice of the data (chunkSize = totalRows / totalRounds), and the last round absorbs the remainder.

4.4 Complete example (package → start monitor → simulate)

# Terminal 1 — package
sh scripts/package-experiment.sh exp-ecg5000-anomaly-001

# Terminal 2 — start the monitor
sh scripts/run-admin-web.sh

In the browser:

  1. http://127.0.0.1:8088/experiments/deploy → upload target/packages/exp-ecg5000-anomaly-001.hdtpkg.zip
  2. Open the deployed experiment → ▶ Session tab
  3. Fill in type=ecg, path=research/datasets/ecg5000/ECG5000, maxClients=5, totalRounds=3
  4. Start session → Send round (or Run All Remaining Rounds)

References: docs/simulation-deployment-guide.md, docs/components/14-iot-replay-simulator.md, docs/execution-guide.md.


Part 5 — Monitoring the experiment

The web monitor (http://127.0.0.1:8088) auto-refreshes every 30 seconds on the dashboard and details screens.

5.1 Dashboard (/)

One card per experiment with: name/objective, contract status badge, counters (acceptances, pending, not accepted, rounds, tasks, assets), and action buttons (Details, Tasks, Performances, Models, Assets, Start round, Pause).

5.2 Experiment details (/experiments/{id})

  • contract header (coordinator, digital twin type, task type, hash);
  • per-client consent table (ACCEPTED/REJECTED/REVOKED/PENDING), with Accept/Reject actions;
  • rounds table: status, accepted count, tasks, containers, examples, local metric, loss, global metric, trend (↑/—/↓), average norm, deadline;
  • learning metrics charts per round (weighted loss, global metric, weighted local metric).

5.3 Round details (/experiments/{id}/rounds/{roundId})

Summary cards (participating clients, examples, weighted local metric, weighted loss, global metric, average/maximum norm, model version), per-client table with preflight failure diagnostics, per-client local convergence (loss/accuracy per epoch, per-class accuracy for HAR — useful for detecting client drift), Docker containers (when applicable), round tasks, and a chronological audit trail of ledger events.

5.4 Other useful pages

Page Route Content
Task plan /experiments/{id}/tasks All tasks (ROUND_CONFIG, LOCAL_TRAINING, PREFLIGHT_CHECK, AGGREGATION, MODEL_PUBLICATION, CONSENT_REGISTRATION) with status and diagnostics
Performances /experiments/{id}/performances Time series of metrics per client/round
Global models /experiments/{id}/models v0 (seed) onward, with hash, URI, and transaction
Hyperledger assets /experiments/{id}/assets Contracts, consents, and audit events on the ledger
Clients /experiments/{id}/clients Consent, dataset evidence, and per-client participation
Blockchain Explorer /experiments/{id}/blockchain All assets by type + chronological events
Configuration /experiments/{id}/config Session hyperparameter editor (rounds, epochs, LR, FedProx μ, dropout, clipping, fine-tuning)

5.5 JSON API (for automation/external dashboards)

curl http://127.0.0.1:8088/api/experiments
curl "http://127.0.0.1:8088/api/experiments/{id}/rounds?version=1.0.0"
curl "http://127.0.0.1:8088/api/experiments/{id}/performances?version=1.0.0"
curl http://127.0.0.1:8088/api/experiments/{id}/models
curl http://127.0.0.1:8088/api/experiments/{id}/sim-session

5.6 What to check after the simulation

  • the experiment appears in the monitor;
  • the session goes from ACTIVE to DONE;
  • rounds are visible on the experiment screen;
  • artifacts appear under target/artifacts/:
data/{client_id}/{experiment_id}/dataset.csv
target/artifacts/{experiment_id}/{client_id}/round-{r}/local_update.bin
target/artifacts/{experiment_id}/{client_id}/round-{r}/metrics.json
target/artifacts/{experiment_id}/round-{r}/aggregation_report.json
target/artifacts/{experiment_id}/models/v{n}/model.bin
  • the 📡 MQTT tab shows traffic when the MQTT gateway is enabled.

References: docs/components/01-admin-api-monitor-web.md, docs/simulation-deployment-guide.md.


Common problems

Symptom Probable cause / check
Packaging fails saying the experiment "has not been migrated yet" experimentId is not registered in the case of scripts/package-experiment.sh — add its own pom.xml and the entry in the script
Monitor comes up, but there is no MQTT traffic Broker inactive, incorrect MQTT_BROKER_URL, or the simulator/experiment did not publish messages
Simulation goes into ERROR Incorrect dataset path, wrong adapter type (har/ecg), MQTT broker unavailable — check the session's last_message and the monitor's logs
Deployed experiment does not appear correctly Upload is not a valid .hdtpkg.zip, sessionFactoryClass not loadable, or SHA256SUMS failed validation
Port 8088 busy The run-admin-web.sh script automatically terminates the previous process via lsof — if it persists, kill the process manually

Environment variables — consolidated reference

Variable Default Description
FEDERATED_LEDGER_MODE blockchain blockchain (only mode supported — HYBRID removed)
FEDERATED_CLIENT_EXECUTION_MODE jvm jvm or docker
HAR_MAX_CLIENTS 30 Limits HAR experiment clients
HDT_AF_DATA_ROOT data Root of per-client data
HDT_AF_ARTIFACT_ROOT target/artifacts Root of off-chain artifacts
HDT_AF_RUNTIME_ROOT target/runtime* Root of temporary runtime
HDT_AF_RESEARCH_ROOT research Root of datasets and specs
SIM_HMAC_SECRET web-sim-secret-dev HMAC secret for web session MQTT envelopes (change in production)
LGPD_PSEUDONYM_SALT — Salt for on-chain pseudonymization
ADMIN_TLS_KEYSTORE / ADMIN_TLS_PASSWORD — Monitor TLS (required for personal data in production)
FABRIC_GATEWAY_PEER_ENDPOINT, FABRIC_CHANNEL_NAME, FABRIC_CHAINCODE_NAME, FABRIC_GATEWAY_CERT_PATH, FABRIC_GATEWAY_KEY_PATH varies Required Hyperledger Fabric Gateway configuration (BLOCKCHAIN only)

Complete project references