Ir para o conteúdo

Execution Guide — HDT-AF

This guide covers the supported build and runtime entry points. The complete module list is in modules.md; the root Maven reactor and its POMs define the active Java modules.


Prerequisites

Tool Requirement Note
Java (JDK) 25 Required by the current root Maven compiler configuration.
Maven Installed and available on PATH Required by the build and test scripts.
Docker Current Docker Compose plugin Needed for broker/Fabric infrastructure and Docker client runs.
lsof, pgrep Recommended Used by the monitor launcher to clean up stale local processes.

The Android application is a separate Gradle build; use its README for Android SDK and Gradle requirements. The project's module checkouts are separate Git repositories; see modules.md before setting up a fresh checkout.

Repository layout

projeto_hdt_af/
├── modules/                 Java modules; each checkout is independently versioned
├── research/experiments/    experiment and benchmark Maven modules
├── modules/scripts/         build, test, run, packaging, and shell-check scripts
├── scripts -> modules/scripts compatibility link used by existing commands
├── deploy/                  Docker, MQTT, and Fabric configuration
├── docs/                    architecture, operation, and component documentation
└── data/                    local per-client/per-experiment data (not source code)

1. Build

The root Maven reactor compiles the platform and its experiment/benchmark modules. Use the script when preparing the runtime classpath for the command-line applications:

sh scripts/compile.sh

The script runs Maven install -DskipTests, collects compiled classes under target/classes, and copies external runtime dependencies into target/lib. A direct mvn compile compiles the reactor but does not perform that runtime-classpath preparation.

2. Integrated test suite

sh scripts/run-tests.sh
# Fast, self-contained unit and integration tags:
sh scripts/run-tests.sh --fast
# One Maven module:
sh scripts/run-tests.sh --module federated-common

The runner first executes the hermetic shell checks in scripts/test-packaging-scripts.sh, then runs Maven/Surefire. The --fast selection avoids the e2e scenarios that need external data or infrastructure. The exact test count is checkout-dependent; use the run's Surefire summary rather than a fixed count in this guide.

Active platform module Main responsibility
federated-common Shared contracts, MQTT envelopes/topics, crypto, codecs, and package utilities
federated-experiment-sdk Experiment API, session lifecycle, runtime paths, and ML descriptors
federated-experiment-packager Experiment, simulation-data, and Android client package formats
federated-local-cdm DataComponent, preflight, verified MQTT ingestion, and SQLite
federated-chaincode Contract, consent, and experiment chaincode domain operations
federated-package-experiment-service Artifact storage, hashing, safe paths, and retention
federated-model-inference Versioned model package and HTTP inference service
federated-client Federated client lifecycle and reusable training engines
federated-orchestrator Round selection/coordination, update relay, and result publication
federated-aggregator Quorum validation and aggregation strategies
federated-containers Paho MQTT 5/TLS gateway and container entry points
federated-digital-twin-broker Local digital-twin broker boundary and validation adapters
federated-rest-api Admin monitor/API, deployment, simulation controls, and integrations
federated-simulator MQTT replay and generic dataset package playback
federated-system-tests Cross-module and end-to-end suite

For the full module and experiment inventory, including separate Gradle/npm projects and inactive legacy directories, see modules.md.


3. Web Monitor (Admin API)

Execution

# Default mode (blockchain — the only mode supported; legacy aliases such as
# memory/hybrid/in-memory/local/both are normalized to blockchain by the script)
sh scripts/run-admin-web.sh

# Explicit blockchain mode
sh scripts/run-admin-web.sh blockchain

# Using an environment variable
FEDERATED_LEDGER_MODE=blockchain sh scripts/run-admin-web.sh

The monitor starts on the fixed port 8088. If the port is busy, the script automatically terminates the previous process (uses lsof). When the .env file exists at the root, the script exports it automatically before selecting the mode and creating the MQTT/Fabric connections.

Access at: http://localhost:8088

Environment variables relevant to blockchain mode

Variable Default Description
FEDERATED_LEDGER_MODE blockchain Ledger mode: blockchain (only mode supported — HYBRID removed)
FABRIC_GATEWAY_PEER_ENDPOINT localhost:7051 gRPC/TLS endpoint of the peer accessed by the Gateway SDK
FABRIC_GATEWAY_CERT_PATH — MSP certificate of the application identity
FABRIC_GATEWAY_KEY_PATH — MSP private key of the application identity
FABRIC_CHANNEL_NAME federated-channel Fabric channel where the chaincodes are installed
FABRIC_CHAINCODE_NAME federated-governance Name of the installed chaincode
FABRIC_FAIL_ON_ERROR true (blockchain default) Whether to throw an exception or just log failures

[!TIP] Para implantação da stack completa (Admin Web + Mosquitto MQTT TLS + PostgreSQL + Hyperledger Fabric) em ambiente servidor/VM via Docker Compose com bridging de dupla rede (hdt-net e fabric_test), consulte o Guia de Implantação em VM e a Topologia de Implantação em VM.


4. Experiment — Industrial Digital Twins (DT)

ID: exp-dt-federated-001

Simulates three industrial clients with digital twins that collaborate to predict equipment failure without centralizing data.

The DT experiment is registered in the monitor and is run through its simulation session. The standalone run-simulator.sh and run-simulator-docker.sh wrappers are not present in this checkout. For a command-line platform smoke run, use sh scripts/run-smoke-experiment.sh.

In monitor Docker mode, each federated participant runs in a separate container. Mounted volumes:

Local volume Container volume Mode
target/classes /app/classes read-only
data/{client_id}/{experiment_id} /app/data/{client_id}/{experiment_id} read-only
target/artifacts /app/artifacts read/write
target/docker-runtime/{exp}/round-{r}/clients/{c} /app/runtime read/write
target/docker-sim/{exp}/round-{r}/clients/{c} /app/output read/write
logs/docker/{exp}/round-{r}/clients/{c} /app/logs read/write

See 06-federated-client.md for the exact, current table (this one has drifted on the /app/data row since BL-091 — the mount is now a filtered timeseries.db file, not a per-client directory).

Environment variable for Docker mode

FEDERATED_CLIENT_EXECUTION_MODE=docker sh scripts/run-admin-web.sh

5. Experiment — HAR with Smartphones (Federated MLP)

ID: exp-har-smartphones-001

The 21 volunteers from the UCI HAR training split are treated as federated clients. An MLP (561→256→128→6) is trained locally by each subject with Adam, FedProx, dropout, L2, gradient clipping, and early stopping, and aggregated via FedAvg. The official test split remains as the researcher's/Aggregator's central dataset to evaluate local models and the global model, without being sent to the clients.

Option 1: automatically downloaded dataset

sh scripts/run-har-experiment.sh

The system downloads the UCI HAR dataset from the official repository and unpacks it into research/datasets/uci-har/.

sh scripts/run-har-experiment.sh "research/datasets/uci-har/UCI HAR Dataset"

The directory must contain:

features.txt
activity_labels.txt
train/
  X_train.txt        ← 7352 rows × 561 features
  y_train.txt        ← 7352 labels (1–6)
  subject_train.txt  ← 7352 subject identifiers (1–30)
test/
  X_test.txt
  y_test.txt
  subject_test.txt

Option 3: limit the number of clients (fast development)

# Only the first 5 subjects of the federated training split participate in the round
HAR_MAX_CLIENTS=5 sh scripts/run-har-experiment.sh "research/datasets/uci-har/UCI HAR Dataset"

With 5 clients, the full run takes about 30 seconds (depending on hardware). With all 21 training clients, it can take several minutes.

Docker execution

sh scripts/run-har-experiment-docker.sh

# With local dataset
sh scripts/run-har-experiment-docker.sh "research/datasets/uci-har/UCI HAR Dataset"

# With client limit
HAR_MAX_CLIENTS=5 sh scripts/run-har-experiment-docker.sh "research/datasets/uci-har/UCI HAR Dataset"

Environment variables — HAR experiment

Variable Default Description
HAR_MAX_CLIENTS 30 Limits participating clients; values above the 21 training subjects use all available clients
HDT_AF_DATA_ROOT data Root of per-client data
HDT_AF_ARTIFACT_ROOT target/artifacts Root of off-chain artifacts
HDT_AF_RUNTIME_ROOT target/runtime* Root of temporary runtime files
HDT_AF_RESEARCH_ROOT research Root of research datasets and specifications
FEDERATED_LEDGER_MODE blockchain Ledger mode (blockchain — only mode supported)

6. Experiment — ECG5000 (Federated Anomaly Detection)

ID: exp-ecg5000-anomaly-001

The official ECG5000_TRAIN split (500 heartbeats) is shuffled with seed=42 and divided across 5 federated clients (~100 samples per client). The official ECG5000_TEST split (4,500 heartbeats) remains as the researcher's central holdout for contractual evaluation of the global model and the local models, without being materialized in data/{client_id}/{experiment_id}. An MLP Autoencoder (140→64→32→16→32→64→140, 23,388 parameters) is trained locally only on normal samples by each client, using Adam + FedProx (μ=0.01) + L2 weight decay. Anomalies are detected via the adaptive threshold mean_error + 2.5·std. The experiment runs 3 federated rounds with DP active (dp_noise_multiplier=1.1 in the default federated phase).

Option 1: automatically downloaded dataset

sh scripts/run-ecg-experiment.sh

The system downloads the ECG5000 dataset from https://timeseriesclassification.com/Downloads/ECG5000.zip and extracts it into research/datasets/ecg5000/.

sh scripts/run-ecg-experiment.sh "research/datasets/ecg5000/ECG5000"

The directory must contain:

ECG5000_TRAIN.arff   ← 500 samples (ARFF format)
ECG5000_TEST.arff    ← 4500 samples (ARFF format)
Or in UCR TXT format:
ECG5000_TRAIN.txt    ← 500 lines: label + 140 space-separated values
ECG5000_TEST.txt     ← 4500 lines

Environment variables — ECG experiment

Variable Default Description
FEDERATED_LEDGER_MODE blockchain Ledger mode (blockchain — only mode supported)
HDT_AF_DATA_ROOT data Root of per-client data
HDT_AF_ARTIFACT_ROOT target/artifacts Root of off-chain artifacts

7. Combined execution: Monitor + Experiment

To follow an experiment in real time on the monitor:

Terminal 1 — starts the monitor:

sh scripts/run-admin-web.sh

Terminal 2 — runs the experiment:

sh scripts/run-har-experiment.sh "research/datasets/uci-har/UCI HAR Dataset"

The monitor reads audited rounds from the required Fabric ledger. Configure the same Fabric channel, chaincode, and application identity for both processes when running them separately.

For separate execution with real-time updates, use the "Start round" button in the web interface — it triggers the executor directly within the monitor's context.


8. Round-by-Round Simulation Session (via Web Monitor)

The simulation session allows running experiments round by round directly in the browser, without a command line.

Terminal — starts the monitor with the integrated simulator:

sh scripts/run-admin-web.sh

In the browser (http://localhost:8088):

  1. Click on an experiment → "▶ Session" tab.
  2. Fill in: adapter type (har or ecg), dataset path, number of clients, and total rounds.
  3. Click Start session.
  4. Use Send round to advance one round at a time, or ⏩ Run All Remaining Rounds.
  5. The experiment details tab updates automatically (auto-refresh every 30s).

Environment variable

Variable Default Description
SIM_HMAC_SECRET "web-sim-secret-dev" HMAC secret for the session's MQTT envelopes. Set a random value in production.

Command-line simulator (IoT MQTT replay)

# Replay UCI HAR via MQTT
sh scripts/run-iot-simulator.sh har

# Replay ECG5000 via MQTT
sh scripts/run-iot-simulator.sh ecg

# Replay digital twins
# The IoT replay runner currently supports har and ecg only.

See docs/components/14-iot-replay-simulator.md for details on round-based slicing and the session's JSON API.


9. Logging and Observability

Structured logs (SLF4J + Logback)

Every run (locally via mvn/scripts, or in a Docker container) writes structured logs to logs/ at the repository root — always the root, never the module/container directory:

logs/hdt-af.log        ← all levels (INFO+), daily/50MB rotation, retained 30 days
logs/hdt-af-error.log  ← WARN/ERROR only, retained 90 days

Line format: 2026-07-27 10:49:35.196 [main] INFO b.u.l.admin.AdminWebServer - [HTTP] GET /health -> 200 (10 ms).

Points with structured business logging (beyond the conversion of every pre-existing System.out/err.println):

Point Format Where
HTTP request [HTTP] {method} {path} -> {status} ({duration}ms) AdminWebServer.handle
OS call [OS-EXEC] {command} -> exitCode={n} ({duration}ms) ProcessExecutor, FederatedDockerClientExecutor
Chaincode transaction [CHAINCODE] {operation} -> {txId} ExperimentContractChaincode, ClientConsentChaincode, FederatedExperimentChaincode
Round execution [ROUND] experimentId={} roundId={} ({duration}ms) AdminExperimentExecutor.execute

Observability stack (Prometheus + Loki + Grafana)

docker compose -f docker-compose.observability.yml up -d
# Grafana:    http://localhost:${GRAFANA_PORT:-3000}   (admin/admin by default)
# Prometheus: http://localhost:${PROMETHEUS_PORT:-9090}

Dedicated compose file (docker-compose.observability.yml, same hdt-af project, standalone — does not depend on mosquitto/postgres/hdt-af-admin being up). docker compose up from the main file does not bring up any of the 5 services (Node Exporter, Prometheus, Promtail, Loki, Grafana); scripts/run-admin-web.sh already brings up this stack automatically when starting the monitor (best-effort — without Docker available, the monitor comes up normally without observability). Host ports configurable via .env (NODE_EXPORTER_PORT/PROMETHEUS_PORT/LOKI_PORT/GRAFANA_PORT) to avoid collisions with other stacks already running on the developer's machine. Grafana already ships with the Prometheus and Loki datasources provisioned.

Docker container log collection does not work on Docker Desktop (macOS/Windows) — only on native Docker Engine (Linux), because Docker Desktop runs containers inside an isolated VM where /var/lib/docker is not visible to the host. In those cases, logs from local executions (logs/*.log) continue to be collected normally. See docker/README.md for the full breakdown, including the alternative via the Loki Docker Logging Driver.

Long-lived connections and the duration panels (BL-197)

SSE connections hold the HTTP handler thread for as long as they live (a BL-140 decision), so the log line for /api/events/stream is only emitted when the connection closes, with a duration equal to the connection's entire lifetime — potentially hours, since it is the dashboard's real-time update channel.

Aggregated with everything else through unwrap duration_ms, a single 6-hour disconnect (21,600,000 ms) skewed the "Duração média" and "p95" panels by orders of magnitude, making the latency dashboard look like it was reporting a catastrophic performance problem when it was just a long SSE session ending normally.

Two layers protect against this, and both are needed:

Layer What it does
[HTTP-STREAM] log marker AdminWebServer logs long-lived streams under a marker that does not contain the substring [HTTP] — Loki's line filter is substring-based, so \|= "[HTTP]" requires the ] right after HTTP and never matches [HTTP-STREAM]. This removes streams from every existing and future [HTTP] aggregation at once, without each query having to remember.
path != "/api/events/stream" in the dashboard Logs already ingested into Loki still carry the route under [HTTP]; without this the historical series stays skewed.

The stream line stays readable and self-describing — it states that the duration is connection lifetime, not latency — so nothing is hidden, it just stops polluting the latency metric.


10. Execution with Hyperledger Fabric (blockchain mode)

The Fabric integration uses the federated-governance chaincode on channel federated-channel. Confirm the deployed chaincode definition and MSP/TLS material for your environment before running. To activate real integration with the blockchain, apply the environment script before starting any component:

source deploy/fabric/fabric-env.sh
sh scripts/run-admin-web.sh

The deploy/fabric/fabric-env.sh script exports the Gateway configuration (FABRIC_GATEWAY_*), the channel/chaincode, and sets FEDERATED_LEDGER_MODE=blockchain — the only supported mode. The monitor uses the Fabric Gateway SDK and requires readable TLS and MSP identity files. The launcher exits with code 78 when required files are absent; legacy mode aliases normalize to blockchain and do not bypass audit.

Prerequisite: test-network running

cd /Users/josedaniel/hyperledger/fabric-samples/test-network
./network.sh up createChannel -c federated-channel -ca

Environment variables — Fabric integration

Variable Default Description
FEDERATED_LEDGER_MODE blockchain blockchain (only mode supported; memory/hybrid/in-memory/local/both are legacy aliases normalized to it)
FABRIC_GATEWAY_PEER_ENDPOINT localhost:7051 gRPC/TLS endpoint of the peer
FABRIC_GATEWAY_TLS_SERVER_NAME_OVERRIDE peer0.org1.example.com Expected name in the peer's TLS certificate
FABRIC_GATEWAY_TLS_CERT_PATH — Peer's TLS root certificate
FABRIC_GATEWAY_MSP_ID Org1MSP MSP of the application identity
FABRIC_GATEWAY_CERT_PATH — X.509 certificate of the MSP identity
FABRIC_GATEWAY_KEY_PATH — Private key of the MSP identity
FABRIC_CHANNEL_NAME federated-channel Fabric channel
FABRIC_CHAINCODE_NAME federated-governance Chaincode name
FABRIC_FAIL_ON_ERROR true Fabric failures block the operation; blockchain is the only supported mode

See docs/blockchain/04-system-integration.md for the full chaincode redeployment procedure.

Fat JAR build (only for chaincode updates)

cd deploy/fabric/chaincode/federated-governance-java
# Requires shadow plugin 7.1.2 — 8.x versions are incompatible with the fabric-javaenv container's Gradle 9.3
gradle shadowJar
# Produces: build/libs/federated-governance-chaincode-0.1.0.jar

11. Outputs generated after execution

Path Content
data/{client_id}/{experiment_id}/dataset.csv Client's partitioned dataset
data/{client_id}/{experiment_id}/metadata.json Dataset metadata (origin, count, provenance)
target/artifacts/{experiment_id}/{client_id}/round-{r}/local_update.bin Local update vector (CSV of doubles)
target/artifacts/{experiment_id}/{client_id}/round-{r}/metrics.json Client training metrics
target/artifacts/{experiment_id}/{client_id}/round-{r}/data_profile.json Authorized data profile
target/artifacts/{experiment_id}/round-{r}/aggregation_report.json Aggregation report with canonical hashes
target/artifacts/{experiment_id}/models/v{n}/model.bin Candidate global model weights
target/classes/ Compiled bytecode (generated by compile.sh)
target/fabric/federated-governance-chaincode.tar.gz Chaincode package for Fabric deployment
logs/hdt-af.log Structured log (SLF4J/Logback) of every run, always at the repository root
logs/hdt-af-error.log WARN/ERROR lines only, same log