Execution Guide — HDT-AF¶
This guide covers the supported build and runtime entry points. The complete module list is in
modules.md; the root Maven reactor and its POMs define the active Java modules.
Prerequisites¶
| Tool | Requirement | Note |
|---|---|---|
| Java (JDK) | 25 | Required by the current root Maven compiler configuration. |
| Maven | Installed and available on PATH |
Required by the build and test scripts. |
| Docker | Current Docker Compose plugin | Needed for broker/Fabric infrastructure and Docker client runs. |
lsof, pgrep |
Recommended | Used by the monitor launcher to clean up stale local processes. |
The Android application is a separate Gradle build; use its README
for Android SDK and Gradle requirements. The project's module checkouts are separate Git repositories;
see modules.md before setting up a fresh checkout.
Repository layout¶
projeto_hdt_af/
├── modules/ Java modules; each checkout is independently versioned
├── research/experiments/ experiment and benchmark Maven modules
├── modules/scripts/ build, test, run, packaging, and shell-check scripts
├── scripts -> modules/scripts compatibility link used by existing commands
├── deploy/ Docker, MQTT, and Fabric configuration
├── docs/ architecture, operation, and component documentation
└── data/ local per-client/per-experiment data (not source code)
1. Build¶
The root Maven reactor compiles the platform and its experiment/benchmark modules. Use the script when preparing the runtime classpath for the command-line applications:
The script runs Maven install -DskipTests, collects compiled classes under target/classes, and
copies external runtime dependencies into target/lib. A direct mvn compile compiles the reactor
but does not perform that runtime-classpath preparation.
2. Integrated test suite¶
sh scripts/run-tests.sh
# Fast, self-contained unit and integration tags:
sh scripts/run-tests.sh --fast
# One Maven module:
sh scripts/run-tests.sh --module federated-common
The runner first executes the hermetic shell checks in scripts/test-packaging-scripts.sh, then
runs Maven/Surefire. The --fast selection avoids the e2e scenarios that need external data or
infrastructure. The exact test count is checkout-dependent; use the run's Surefire summary rather
than a fixed count in this guide.
| Active platform module | Main responsibility |
|---|---|
federated-common |
Shared contracts, MQTT envelopes/topics, crypto, codecs, and package utilities |
federated-experiment-sdk |
Experiment API, session lifecycle, runtime paths, and ML descriptors |
federated-experiment-packager |
Experiment, simulation-data, and Android client package formats |
federated-local-cdm |
DataComponent, preflight, verified MQTT ingestion, and SQLite |
federated-chaincode |
Contract, consent, and experiment chaincode domain operations |
federated-package-experiment-service |
Artifact storage, hashing, safe paths, and retention |
federated-model-inference |
Versioned model package and HTTP inference service |
federated-client |
Federated client lifecycle and reusable training engines |
federated-orchestrator |
Round selection/coordination, update relay, and result publication |
federated-aggregator |
Quorum validation and aggregation strategies |
federated-containers |
Paho MQTT 5/TLS gateway and container entry points |
federated-digital-twin-broker |
Local digital-twin broker boundary and validation adapters |
federated-rest-api |
Admin monitor/API, deployment, simulation controls, and integrations |
federated-simulator |
MQTT replay and generic dataset package playback |
federated-system-tests |
Cross-module and end-to-end suite |
For the full module and experiment inventory, including separate Gradle/npm projects and inactive
legacy directories, see modules.md.
3. Web Monitor (Admin API)¶
Execution¶
# Default mode (blockchain — the only mode supported; legacy aliases such as
# memory/hybrid/in-memory/local/both are normalized to blockchain by the script)
sh scripts/run-admin-web.sh
# Explicit blockchain mode
sh scripts/run-admin-web.sh blockchain
# Using an environment variable
FEDERATED_LEDGER_MODE=blockchain sh scripts/run-admin-web.sh
The monitor starts on the fixed port 8088. If the port is busy, the script automatically terminates the previous process (uses lsof). When the .env file exists at the root, the script exports it automatically before selecting the mode and creating the MQTT/Fabric connections.
Access at: http://localhost:8088
Environment variables relevant to blockchain mode¶
| Variable | Default | Description |
|---|---|---|
FEDERATED_LEDGER_MODE |
blockchain |
Ledger mode: blockchain (only mode supported — HYBRID removed) |
FABRIC_GATEWAY_PEER_ENDPOINT |
localhost:7051 |
gRPC/TLS endpoint of the peer accessed by the Gateway SDK |
FABRIC_GATEWAY_CERT_PATH |
— | MSP certificate of the application identity |
FABRIC_GATEWAY_KEY_PATH |
— | MSP private key of the application identity |
FABRIC_CHANNEL_NAME |
federated-channel |
Fabric channel where the chaincodes are installed |
FABRIC_CHAINCODE_NAME |
federated-governance |
Name of the installed chaincode |
FABRIC_FAIL_ON_ERROR |
true (blockchain default) |
Whether to throw an exception or just log failures |
[!TIP] Para implantação da stack completa (Admin Web + Mosquitto MQTT TLS + PostgreSQL + Hyperledger Fabric) em ambiente servidor/VM via Docker Compose com bridging de dupla rede (
hdt-netefabric_test), consulte o Guia de Implantação em VM e a Topologia de Implantação em VM.
4. Experiment — Industrial Digital Twins (DT)¶
ID: exp-dt-federated-001
Simulates three industrial clients with digital twins that collaborate to predict equipment failure without centralizing data.
The DT experiment is registered in the monitor and is run through its simulation session. The
standalone run-simulator.sh and run-simulator-docker.sh wrappers are not present in this
checkout. For a command-line platform smoke run, use sh scripts/run-smoke-experiment.sh.
In monitor Docker mode, each federated participant runs in a separate container. Mounted volumes:
| Local volume | Container volume | Mode |
|---|---|---|
target/classes |
/app/classes |
read-only |
data/{client_id}/{experiment_id} |
/app/data/{client_id}/{experiment_id} |
read-only |
target/artifacts |
/app/artifacts |
read/write |
target/docker-runtime/{exp}/round-{r}/clients/{c} |
/app/runtime |
read/write |
target/docker-sim/{exp}/round-{r}/clients/{c} |
/app/output |
read/write |
logs/docker/{exp}/round-{r}/clients/{c} |
/app/logs |
read/write |
See
06-federated-client.mdfor the exact, current table (this one has drifted on the/app/datarow since BL-091 — the mount is now a filteredtimeseries.dbfile, not a per-client directory).
Environment variable for Docker mode¶
5. Experiment — HAR with Smartphones (Federated MLP)¶
ID: exp-har-smartphones-001
The 21 volunteers from the UCI HAR training split are treated as federated clients. An MLP (561→256→128→6) is trained locally by each subject with Adam, FedProx, dropout, L2, gradient clipping, and early stopping, and aggregated via FedAvg. The official test split remains as the researcher's/Aggregator's central dataset to evaluate local models and the global model, without being sent to the clients.
Option 1: automatically downloaded dataset¶
The system downloads the UCI HAR dataset from the official repository and unpacks it into research/datasets/uci-har/.
Option 2: local dataset already available (recommended)¶
The directory must contain:
features.txt
activity_labels.txt
train/
X_train.txt ← 7352 rows × 561 features
y_train.txt ← 7352 labels (1–6)
subject_train.txt ← 7352 subject identifiers (1–30)
test/
X_test.txt
y_test.txt
subject_test.txt
Option 3: limit the number of clients (fast development)¶
# Only the first 5 subjects of the federated training split participate in the round
HAR_MAX_CLIENTS=5 sh scripts/run-har-experiment.sh "research/datasets/uci-har/UCI HAR Dataset"
With 5 clients, the full run takes about 30 seconds (depending on hardware). With all 21 training clients, it can take several minutes.
Docker execution¶
sh scripts/run-har-experiment-docker.sh
# With local dataset
sh scripts/run-har-experiment-docker.sh "research/datasets/uci-har/UCI HAR Dataset"
# With client limit
HAR_MAX_CLIENTS=5 sh scripts/run-har-experiment-docker.sh "research/datasets/uci-har/UCI HAR Dataset"
Environment variables — HAR experiment¶
| Variable | Default | Description |
|---|---|---|
HAR_MAX_CLIENTS |
30 |
Limits participating clients; values above the 21 training subjects use all available clients |
HDT_AF_DATA_ROOT |
data |
Root of per-client data |
HDT_AF_ARTIFACT_ROOT |
target/artifacts |
Root of off-chain artifacts |
HDT_AF_RUNTIME_ROOT |
target/runtime* |
Root of temporary runtime files |
HDT_AF_RESEARCH_ROOT |
research |
Root of research datasets and specifications |
FEDERATED_LEDGER_MODE |
blockchain |
Ledger mode (blockchain — only mode supported) |
6. Experiment — ECG5000 (Federated Anomaly Detection)¶
ID: exp-ecg5000-anomaly-001
The official ECG5000_TRAIN split (500 heartbeats) is shuffled with seed=42 and divided across 5 federated clients (~100 samples per client). The official ECG5000_TEST split (4,500 heartbeats) remains as the researcher's central holdout for contractual evaluation of the global model and the local models, without being materialized in data/{client_id}/{experiment_id}. An MLP Autoencoder (140→64→32→16→32→64→140, 23,388 parameters) is trained locally only on normal samples by each client, using Adam + FedProx (μ=0.01) + L2 weight decay. Anomalies are detected via the adaptive threshold mean_error + 2.5·std. The experiment runs 3 federated rounds with DP active (dp_noise_multiplier=1.1 in the default federated phase).
Option 1: automatically downloaded dataset¶
The system downloads the ECG5000 dataset from https://timeseriesclassification.com/Downloads/ECG5000.zip and extracts it into research/datasets/ecg5000/.
Option 2: local dataset already available (recommended)¶
The directory must contain:
Or in UCR TXT format:Environment variables — ECG experiment¶
| Variable | Default | Description |
|---|---|---|
FEDERATED_LEDGER_MODE |
blockchain |
Ledger mode (blockchain — only mode supported) |
HDT_AF_DATA_ROOT |
data |
Root of per-client data |
HDT_AF_ARTIFACT_ROOT |
target/artifacts |
Root of off-chain artifacts |
7. Combined execution: Monitor + Experiment¶
To follow an experiment in real time on the monitor:
Terminal 1 — starts the monitor:
Terminal 2 — runs the experiment:
The monitor reads audited rounds from the required Fabric ledger. Configure the same Fabric channel, chaincode, and application identity for both processes when running them separately.
For separate execution with real-time updates, use the "Start round" button in the web interface — it triggers the executor directly within the monitor's context.
8. Round-by-Round Simulation Session (via Web Monitor)¶
The simulation session allows running experiments round by round directly in the browser, without a command line.
Terminal — starts the monitor with the integrated simulator:
In the browser (http://localhost:8088):
- Click on an experiment → "▶ Session" tab.
- Fill in: adapter type (
harorecg), dataset path, number of clients, and total rounds. - Click Start session.
- Use Send round to advance one round at a time, or ⏩ Run All Remaining Rounds.
- The experiment details tab updates automatically (auto-refresh every 30s).
Environment variable¶
| Variable | Default | Description |
|---|---|---|
SIM_HMAC_SECRET |
"web-sim-secret-dev" |
HMAC secret for the session's MQTT envelopes. Set a random value in production. |
Command-line simulator (IoT MQTT replay)¶
# Replay UCI HAR via MQTT
sh scripts/run-iot-simulator.sh har
# Replay ECG5000 via MQTT
sh scripts/run-iot-simulator.sh ecg
# Replay digital twins
# The IoT replay runner currently supports har and ecg only.
See docs/components/14-iot-replay-simulator.md for details on round-based slicing and the session's JSON API.
9. Logging and Observability¶
Structured logs (SLF4J + Logback)¶
Every run (locally via mvn/scripts, or in a Docker container) writes structured logs to
logs/ at the repository root — always the root, never the module/container directory:
logs/hdt-af.log ← all levels (INFO+), daily/50MB rotation, retained 30 days
logs/hdt-af-error.log ← WARN/ERROR only, retained 90 days
Line format: 2026-07-27 10:49:35.196 [main] INFO b.u.l.admin.AdminWebServer - [HTTP] GET /health -> 200 (10 ms).
Points with structured business logging (beyond the conversion of every pre-existing
System.out/err.println):
| Point | Format | Where |
|---|---|---|
| HTTP request | [HTTP] {method} {path} -> {status} ({duration}ms) |
AdminWebServer.handle |
| OS call | [OS-EXEC] {command} -> exitCode={n} ({duration}ms) |
ProcessExecutor, FederatedDockerClientExecutor |
| Chaincode transaction | [CHAINCODE] {operation} -> {txId} |
ExperimentContractChaincode, ClientConsentChaincode, FederatedExperimentChaincode |
| Round execution | [ROUND] experimentId={} roundId={} ({duration}ms) |
AdminExperimentExecutor.execute |
Observability stack (Prometheus + Loki + Grafana)¶
docker compose -f docker-compose.observability.yml up -d
# Grafana: http://localhost:${GRAFANA_PORT:-3000} (admin/admin by default)
# Prometheus: http://localhost:${PROMETHEUS_PORT:-9090}
Dedicated compose file (docker-compose.observability.yml, same hdt-af project, standalone
— does not depend on mosquitto/postgres/hdt-af-admin being up). docker compose up from
the main file does not bring up any of the 5 services (Node Exporter, Prometheus, Promtail, Loki,
Grafana); scripts/run-admin-web.sh already brings up this stack automatically when starting the
monitor (best-effort — without Docker available, the monitor comes up normally without
observability). Host ports configurable via .env (NODE_EXPORTER_PORT/PROMETHEUS_PORT/LOKI_PORT/GRAFANA_PORT)
to avoid collisions with other stacks already running on the developer's machine. Grafana already
ships with the Prometheus and Loki datasources provisioned.
Docker container log collection does not work on Docker Desktop (macOS/Windows) — only on
native Docker Engine (Linux), because Docker Desktop runs containers inside an isolated VM where
/var/lib/docker is not visible to the host. In those cases, logs from local executions
(logs/*.log) continue to be collected normally. See docker/README.md for the full breakdown,
including the alternative via the Loki Docker Logging Driver.
Long-lived connections and the duration panels (BL-197)¶
SSE connections hold the HTTP handler thread for as long as they live (a BL-140 decision), so the
log line for /api/events/stream is only emitted when the connection closes, with a duration
equal to the connection's entire lifetime — potentially hours, since it is the dashboard's
real-time update channel.
Aggregated with everything else through unwrap duration_ms, a single 6-hour disconnect
(21,600,000 ms) skewed the "Duração média" and "p95" panels by orders of magnitude, making the
latency dashboard look like it was reporting a catastrophic performance problem when it was just a
long SSE session ending normally.
Two layers protect against this, and both are needed:
| Layer | What it does |
|---|---|
[HTTP-STREAM] log marker |
AdminWebServer logs long-lived streams under a marker that does not contain the substring [HTTP] — Loki's line filter is substring-based, so \|= "[HTTP]" requires the ] right after HTTP and never matches [HTTP-STREAM]. This removes streams from every existing and future [HTTP] aggregation at once, without each query having to remember. |
path != "/api/events/stream" in the dashboard |
Logs already ingested into Loki still carry the route under [HTTP]; without this the historical series stays skewed. |
The stream line stays readable and self-describing — it states that the duration is connection lifetime, not latency — so nothing is hidden, it just stops polluting the latency metric.
10. Execution with Hyperledger Fabric (blockchain mode)¶
The Fabric integration uses the federated-governance chaincode on channel federated-channel. Confirm the deployed chaincode definition and MSP/TLS material for your environment before running. To activate real integration with the blockchain, apply the environment script before starting any component:
The deploy/fabric/fabric-env.sh script exports the Gateway configuration (FABRIC_GATEWAY_*),
the channel/chaincode, and sets FEDERATED_LEDGER_MODE=blockchain — the only supported mode. The
monitor uses the Fabric Gateway SDK and requires readable TLS and MSP identity files. The launcher
exits with code 78 when required files are absent; legacy mode aliases normalize to blockchain
and do not bypass audit.
Prerequisite: test-network running¶
cd /Users/josedaniel/hyperledger/fabric-samples/test-network
./network.sh up createChannel -c federated-channel -ca
Environment variables — Fabric integration¶
| Variable | Default | Description |
|---|---|---|
FEDERATED_LEDGER_MODE |
blockchain |
blockchain (only mode supported; memory/hybrid/in-memory/local/both are legacy aliases normalized to it) |
FABRIC_GATEWAY_PEER_ENDPOINT |
localhost:7051 |
gRPC/TLS endpoint of the peer |
FABRIC_GATEWAY_TLS_SERVER_NAME_OVERRIDE |
peer0.org1.example.com |
Expected name in the peer's TLS certificate |
FABRIC_GATEWAY_TLS_CERT_PATH |
— | Peer's TLS root certificate |
FABRIC_GATEWAY_MSP_ID |
Org1MSP |
MSP of the application identity |
FABRIC_GATEWAY_CERT_PATH |
— | X.509 certificate of the MSP identity |
FABRIC_GATEWAY_KEY_PATH |
— | Private key of the MSP identity |
FABRIC_CHANNEL_NAME |
federated-channel |
Fabric channel |
FABRIC_CHAINCODE_NAME |
federated-governance |
Chaincode name |
FABRIC_FAIL_ON_ERROR |
true |
Fabric failures block the operation; blockchain is the only supported mode |
See docs/blockchain/04-system-integration.md for the full chaincode redeployment procedure.
Fat JAR build (only for chaincode updates)¶
cd deploy/fabric/chaincode/federated-governance-java
# Requires shadow plugin 7.1.2 — 8.x versions are incompatible with the fabric-javaenv container's Gradle 9.3
gradle shadowJar
# Produces: build/libs/federated-governance-chaincode-0.1.0.jar
11. Outputs generated after execution¶
| Path | Content |
|---|---|
data/{client_id}/{experiment_id}/dataset.csv |
Client's partitioned dataset |
data/{client_id}/{experiment_id}/metadata.json |
Dataset metadata (origin, count, provenance) |
target/artifacts/{experiment_id}/{client_id}/round-{r}/local_update.bin |
Local update vector (CSV of doubles) |
target/artifacts/{experiment_id}/{client_id}/round-{r}/metrics.json |
Client training metrics |
target/artifacts/{experiment_id}/{client_id}/round-{r}/data_profile.json |
Authorized data profile |
target/artifacts/{experiment_id}/round-{r}/aggregation_report.json |
Aggregation report with canonical hashes |
target/artifacts/{experiment_id}/models/v{n}/model.bin |
Candidate global model weights |
target/classes/ |
Compiled bytecode (generated by compile.sh) |
target/fabric/federated-governance-chaincode.tar.gz |
Chaincode package for Fabric deployment |
logs/hdt-af.log |
Structured log (SLF4J/Logback) of every run, always at the repository root |
logs/hdt-af-error.log |
WARN/ERROR lines only, same log |