Ir para o conteúdo

Component: Local Digital Twin Broker (ADR-002)

Responsibility

The local access interface to the digital twin, requested to let the orchestrator send and receive commands/data through a dedicated component instead of the federated client being reachable only via Docker/env vars. Implemented as an embedded broker (BL-238; HTTP+SSE since BL-243, originally an MQTT microbroker) — the same architectural pattern already documented for the external reference project (SESI Ergotech) in docs/cddl_cdal.md/docs/mqtt/MQTT_TOPICS_CDDL.md: a local broker running inside the client's own process/container, serving as an internal bus, bridged to the real external broker through a separate connection.

Closes the gap left open in 12-local-digital-twin.md: the LocalModelValidator sketched there as "next implementation step" is implemented here.

BL-241 update (Phase 1 of BL-240), reverted (BL-259): the external hop of DigitalTwinBridge had briefly stopped being a single MQTT gateway — RoundConfig/ RoundFinalizationSignal/GlobalModelPublication/ClientUpdateMessage moved to REST+SSE while only TwinValidationDecisionMessage stayed on MQTT. That split was reverted end to end: the external hop is back to a single MQTT externalGateway (in production, PahoMqttGateway) for every payload type, TwinValidationDecisionMessage included (it never actually left MQTT). The DigitalTwinBridge constructor now takes one externalGateway again; a deprecated 5-argument overload (externalMqttGateway, externalRestSseGateway) is kept temporarily for any caller not yet updated, and simply ignores the second gateway. The internal hop (embedded broker, BL-243) is untouched by this revert — it still runs over EmbeddedRestSseBroker/RestSseGateway (module federated-digital-twin-broker), described in full further down. Only the external boundary reverted; see "Architecture" below for the updated diagram.

BL-242 — no code change here, composes as-is. DigitalTwinBridge.relayOutbound still delivers ClientUpdateMessage onto the external client-facing topic (MqttTopics.allClientUpdates(experimentId)) — exactly the topic the orchestrator's new ClientUpdateRelay subscribes to. The bridge has no notion of, and no dependency on, the aggregator-facing topic the relay republishes to. See 04-orchestrator.md.

BL-243 — the internal hop also migrated, from MQTT to HTTP+SSE. The embedded MQTT microbroker (Moquette) and its dedicated EmbeddedTwinMqttGateway (MQTT 3.1.1 via Paho v3) are gone. EmbeddedTwinBroker now delegates to EmbeddedRestSseBroker (new production class, module federated-digital-twin-broker) — the same mechanism already used for the external hop since BL-241 — and the bridge/adapters connect to it via plain RestSseGateway instances instead of a Paho v3 client. The Moquette-specific limitation that originally motivated the MQTT 3.1.1 workaround (see "Finding" below, now historical) no longer applies: there is no MQTT anywhere in this module's internal hop. DigitalTwinBridge/DigitalTwinClientCommandAdapter/ DigitalTwinValidatorAdapter/EnvelopeBoundaryGuard needed zero changes — they only ever depended on the MqttGateway interface, never on the concrete transport.


Implementation

Item Value
Module modules/federated-digital-twin-broker
Java package br.ufma.lsdi.digitaltwin (validator classes in br.ufma.lsdi.digitaltwin.validator)
Embedded broker EmbeddedTwinBroker delegates to EmbeddedRestSseBroker (module federated-digital-twin-broker, HTTP+SSE, com.sun.net.httpserver.HttpServer — BL-243; originally Moquette/MQTT 3.1.1, see finding below)
Composition root LocalDigitalTwinRuntime

Architecture

┌─ MQTT (external, reverted to single gateway BL-259) ────────────────────┐
│ External MQTT broker (Mosquitto), ssl://mosquitto:8883, PahoMqttGateway  │
└──────────────────────────────────┬───────────────────────────────────────┘
                                    │ roundConfig · roundFinalization · modelPublished ·
                                    │ clientUpdate (outbound) · twinValidationDecision (outbound)
                                    ▼
              ┌──────────────────────────── DigitalTwinBridge ─────────────────────────┐
              │              externalGateway (single MQTT gateway, all types)           │
              │           verify() + verifyPayloadIntegrity() at the boundary            │
              └────────────────────────────────┬────────────────────────────────────────┘
                                                 │ http://127.0.0.1:{port} — REST+SSE (BL-243,
                                                 │ internal hop only, NOT reverted),
                                                 │ same MqttTopics.* strings as topic/topicFilter
                                                 ▼
              ┌──────────────────────────────────────────────────┐
              │  EmbeddedTwinBroker → EmbeddedRestSseBroker (loopback) │
              │  runs inside the client's process/container            │
              └──────────┬─────────────────┬───────────────────────┘
                          ▼                 ▼
        DigitalTwinClientCommandAdapter   DigitalTwinValidatorAdapter
        - holds RoundConfig pending        - receives GlobalModelPublication
        - on finalization, calls           - calls LocalModelValidator.validate()
          FederatedClientService           - publishes TwinValidationDecisionMessage back out
          .executeRoundOnFinalizationSignal   through the bridge (external MQTT 5, BL-259)
                                             - registers via its OWN local FederatedExperimentChaincode
                                               (in-memory only here — never the real ledger; see
                                               TwinValidationDecisionIngestor, orchestrator side)

Everything below "internalGateway" (the embedded broker and both adapters) is unchanged in behavior by BL-241/BL-243/BL-259 — only the transport carrying the bytes on the external hop moved and then moved back (BL-241 split it into MQTT+REST+SSE; BL-259 reverted it to a single MQTT gateway); the internal hop stayed HTTP+SSE since BL-243 throughout, untouched by the revert. DigitalTwinBridge, both adapters, and EnvelopeBoundaryGuard only ever depend on the MqttGateway interface — none of them needed a single line changed for any of these transport changes.

Topic namespace: the embedded broker reuses the exact same MqttTopics.* strings as the external broker — they are physically distinct endpoints, so there is no collision, and FederatedClientService (which already builds topics via MqttTopics.clientUpdate(...) internally) works unmodified when handed the internal gateway. Since BL-243 those strings are used as topic=/topicFilter= query params over HTTP for the internal hop; the external hop uses them as literal MQTT topic strings again (as it always had, apart from the reverted BL-241 window).

Reuse, no changes to federated-client: FederatedClientService already exposes executeRoundEnvelopeAndPublish(...) and executeRoundOnFinalizationSignal(...), both accepting an arbitrary MqttGateway to publish the resulting ClientUpdateMessage back. This module only feeds them; federated-client itself was not touched.

Data-sovereignty update — no longer just coexistence. The note below ("coexistence, not migration") described this module's status before the fronteira de soberania de dados refactor: FederatedClientService used to write/read the Artifact Experiment Service directly, bypassing this broker entirely. It no longer can — its constructor now takes a ClientArtifactChannel, and every real production/research composition root (*FederatedSession, DockerErgonomicClientDaemonApp, DockerSmokeClientDaemonApp) wires a real EmbeddedTwinBroker + DigitalTwinBridge and a TwinArtifactChannel built on it. The one remaining exception is the ephemeral, host-orchestrated Docker path (Docker*ClientApp, e.g. DockerHarClientApp) — it has no live orchestrator/broker counterpart reachable from inside a single short-lived container, so it uses LocalDiskArtifactChannel instead (module federated-client, see its Javadoc). See "ClientArtifactPayload / GlobalModelPayload / TwinArtifactChannel" below.

Coexistence, not migration (historical — see the update box above): FederatedDockerClientExecutor and the Docker/env-vars path are untouched. This module was, at the time, an additional path.


Historical finding: Moquette 0.17 (Maven Central) only spoke MQTT 3.1.1

Superseded by BL-243 — kept here only as the historical rationale for why the internal hop briefly needed a second MQTT client stack, now removed entirely.

Moquette added MQTT5 support in release 0.18 — but 0.18 was never published to Maven Central (only 0.15/0.16/0.17 are there). PahoMqttGateway, the project's real MqttGateway implementation, is MQTT5-only (org.eclipse.paho.mqttv5.client) and Moquette 0.17 rejected its CONNECT packet ("MQTT protocol version is not valid").

Consequence at the time: the internal hop (bridge/adapters ↔ embedded broker) used a dedicated EmbeddedTwinMqttGateway, built on the MQTT 3.1.1 client (org.eclipse.paho.client.mqttv3, same 1.2.5 release line as the v5 client used elsewhere). BL-243 removed both the Moquette broker and EmbeddedTwinMqttGateway — the internal hop is HTTP+SSE now, so this MQTT-version incompatibility no longer applies to any part of this module.


Main classes

EmbeddedTwinBroker

BL-243: thin wrapper delegating to EmbeddedRestSseBroker (module federated-digital-twin-broker) — no longer encapsulates a Moquette Server. Always binds to 127.0.0.1 (never 0.0.0.0) — the broker is strictly local to the client's own process/container, never exposed externally.

Port resolution: EmbeddedRestSseBroker reserves the port with a plain java.net.ServerSocket bind/close first, same technique the original Moquette-backed class used and for the same reason — empirically, neither com.sun.net.httpserver.HttpServer.create(...) nor .start() reliably throw when the requested port is already occupied (confirmed by direct testing before writing this class), so a synchronous preflight probe is still required, not a Moquette-specific quirk. This also resolves ephemeral ports (port=0) for tests. Small TOCTOU window between the reservation and the real bind; acceptable in tests (re-runnable), production should use a fixed port via DIGITAL_TWIN_BROKER_PORT.

EmbeddedTwinMqttGateway (the old MQTT 3.1.1 Paho-based internal gateway) is removed — the bridge/adapters connect via plain RestSseGateway (module federated-digital-twin-broker) instances instead, the same class already in production for the external hop since BL-241. No TLS/authentication on the internal broker: loopback-only.

EnvelopeBoundaryGuard / BoundaryVerdict

Single point of envelope.verify(signatureService) + envelope.verifyPayloadIntegrity() at the external→internal boundary (per .agents/shared-rules.md). Reused by DigitalTwinBridge, DigitalTwinClientCommandAdapter, and DigitalTwinValidatorAdapter — every receiver re-checks, defense-in-depth, rather than trusting an upstream check.

DigitalTwinBridge

Relays external↔internal in both directions: RoundConfig/RoundFinalizationSignal/ GlobalModelPublication inbound (validated); ClientUpdateMessage/TwinValidationDecisionMessage outbound (already signed by their producer, no re-verification needed). A rejected/malformed envelope is logged and dropped — never propagates and never blocks subsequent messages.

Single external gateway again (BL-241 dual-gateway split reverted, BL-259): the constructor takes one externalGateway (in production, PahoMqttGateway). start() wires all inbound relays (RoundConfig/RoundFinalizationSignal/GlobalModelPublication/GlobalModelPayload) and all outbound relays (ClientUpdateMessage/TwinValidationDecisionMessage/ClientArtifactPayload) to that same gateway. relayInbound/relayOutbound read the fixed externalGateway field again (no gateway parameter). A deprecated 5-argument constructor overload (externalMqttGateway, externalRestSseGateway) is kept temporarily for any not-yet-updated caller — it discards the second gateway and delegates to the single-gateway constructor using the first.

DigitalTwinClientCommandAdapter

Holds a RoundConfig pending only after it passes EnvelopeBoundaryGuard (executeRoundOnFinalizationSignal does not re-verify the RoundConfig's own signature — only the finalization signal's — so this check is this adapter's responsibility, not redundant). On a matching RoundFinalizationSignal, removes the pending config and calls FederatedClientService.executeRoundOnFinalizationSignal(...); failures are caught and logged, never propagated from the MQTT callback.

validator.LocalModelValidator / BaselineComparisonModelValidator

See 12-local-digital-twin.md for the full step-by-step. TwinValidationOutcome (decision + evaluated metric + candidate hash + reason) is a deliberate superset of the original sketch's bare ValidationDecision return — the adapter needs those details to build TwinValidationDecisionMessage and the audit trail.

LocalModelBaseline/PrivateHoldout/HoldoutEvaluator/LocalModelValidator never travel over the wire at all (by the same invariant as the rest of the digital twin — validation holdout stays local), so they live in this module, not in federated-common. ValidationDecision and TwinValidationDecisionMessage do travel as an MqttEnvelope payload (external MQTT 5 after BL-259 revert, and internal HTTP+SSE loopback), so they live in federated-common alongside every other payload type.

DigitalTwinValidatorAdapter

Subscribes to model publications on the internal broker, runs LocalModelValidator, signs and publishes TwinValidationDecisionMessage back out through the internal gateway (from where DigitalTwinBridge relays it externally via MQTT 5), and calls FederatedExperimentChaincode.registerTwinValidationDecision(...) on its own, unbridged chaincode instance — an in-memory-only local audit trail, never the real ledger (mirrors the same invariant established for registerLocalUpdateHash/consent in BL-257: the client/twin never writes to Fabric directly). The authentic registration happens on the orchestrator side, once TwinValidationDecisionIngestor receives the relayed message (BL-259, see below).

LocalDigitalTwinRuntime

Composition root — starts the embedded broker, opens one RestSseGateway per component (bridge/client-adapter/validator-adapter/artifact channel, each with a distinct client ID suffix — no longer a correctness requirement since REST+SSE connections don't collide by client ID the way MQTT sessions did in BL-115, kept only for log/debugging clarity), wires everything, and closes it all in reverse order. Exposes a package-private internalConnectionCount() for test synchronization: RestSseGateway.subscribe() is asynchronous (dedicated connection thread per call), unlike the old EmbeddedTwinMqttGateway.subscribe() (blocking, synchronous Paho SUBACK) — a caller that publishes immediately after construction can race the internal SSE connections still coming up.

Constructor takes a client factory, not a ready client. FederatedClientService needs a TwinArtifactChannel built on the broker's own internal gateway — which only exists once the broker has already started inside this constructor — so LocalDigitalTwinRuntime takes a Function<MqttGateway, FederatedClientService> clientServiceFactory instead of a pre-built FederatedClientService. It opens the artifact channel's internal connection first, then calls the factory with it, then wires the client command adapter with the result.

ClientArtifactPayload / GlobalModelPayload / TwinArtifactChannel

Fronteira de soberania de dados: two new MqttEnvelope payload types (module federated-common) carry the actual artifact bytes (Base64) over this same bridge — ClientArtifactPayload outbound (client → orchestrator: local update/metrics/profile) and GlobalModelPayload inbound (orchestrator → client: the FedProx reference model), both added to DigitalTwinBridge.start() with one relayOutbound/relayInbound line each, no other change to the bridge.

TwinArtifactChannel (module federated-client) is the production ClientArtifactChannel implementation: upload() computes the hash/length locally (pure function, no disk access) and publishes a signed ClientArtifactPayload on the client-facing artifact topic; downloadGlobalModel() subscribes, at construction, to the global-model-payload topic and serves from an in-memory cache, waiting up to a bounded timeout (default 5s) for the payload to arrive — a hash mismatch on an already-received payload is a hard UPDATE_REJECTED (integrity violated), distinct from "not arrived yet" (Optional.empty(), tolerated as pure FedAvg). On the orchestrator side, ClientArtifactIngestor (module federated-orchestrator) subscribes to the same client-facing artifact topic and writes the decoded, hash-verified bytes into its own Artifact Experiment Service, under the same namespace/filename convention ClientUpdateResolver uses to read them back — the client never learns, or needs to know, the real storage path.

Body-size limit raised for real payloads (256 KiB → 8 MiB) & configuration. EmbeddedRestSseBroker defaults to DEFAULT_MAX_BODY_BYTES = 8 MiB, DEFAULT_MAX_CONNECTIONS = 16, a 15s SSE : heartbeat, and also answers 200 on catch-all / so RestSseGateway's eager constructor probe succeeds. Configuration via EmbeddedTwinBrokerConfig.fromEnvironment() (DIGITAL_TWIN_BROKER_PORT default 0, DIGITAL_TWIN_BROKER_HOST default 127.0.0.1 — note bindHost checks non-blank rather than restricting non-loopback in code) and RestSseGatewayConfig.fromEnvironment() (REST_SSE_GATEWAY_URL, REST_SSE_CLIENT_ID, REST_SSE_CONNECT_TIMEOUT_MS); RestSseGateway retries failed SSE connections with exponential backoff (500ms → 15s). (Note: BaselineComparisonModelValidator still accesses LocalArtifactExperimentService directly for model.bin and writes candidate-v{version}.bin, rather than reading GlobalModelPayload in memory.)


Ledger — registerTwinValidationDecision

Added to FederatedExperimentChaincode (in-memory audit event, mirrors the pattern of registerPreflightHash/registerLocalUpdateHash) and, as default no-op methods, to both FabricExperimentBridge and LedgerPort for backward compatibility with existing implementers that don't need this method (e.g. TestLedgerPort).

BL-259: wired to a real Fabric transaction. FederatedExperimentSmartContract (the chaincode in deploy/fabric/chaincode/federated-governance-java) gained a registerTwinValidationDecision transaction, mirroring registerLocalUpdateHash's shape (appendAudit with the decision as metadata). FabricCliLedgerMirror/FabricGatewayLedgerMirror no longer override this method as a no-op — they invoke the transaction for real (pseudonymizing clientId first, same as every other per-client ledger write). New TwinValidationDecisionIngestor (module federated-orchestrator, package br.ufma.lsdi.orchestrator) is the actual subscriber: it receives the message over the FederatedExperimentChaincode.registerTwinValidationDecision(...) on its own chaincode instance, the same one AggregationResultPublisher already uses for registerAggregationHash, bridged to the real network when the process is constructed with one (ledgerMode=BLOCKCHAIN). (TwinValidationDecisionIngestor consumes the decision over the external MQTT 5 gateway.)

Still not validated end-to-end against a running Fabric network in this environment — installing this new chaincode version requires the peer CLI and a Java chaincode builder available on the test-network, neither present here (same limitation as BL-235). The chaincode source itself compiles and its own Gradle test suite passes; TwinValidationDecisionIngestorTest (4 cases, InMemoryMqttGateway + the real FederatedExperimentChaincode, unbridged) proves the Java-side wiring is correct up to the chaincode boundary.

Wired so far only in ErgonomicRiskFederatedSession (the same pilot session BL-234's X.509 migration used first) — propagating to the other 4 canonical *FederatedSession classes and the Docker daemon apps is a registered follow-up, not silently incomplete.


Mandatory rules

  • The embedded broker's bindHost must stay loopback-only (127.0.0.1 by default — note EmbeddedTwinBrokerConfig validates non-blank only, so operators must avoid setting 0.0.0.0 or routable addresses).
  • Every envelope crossing external→internal must pass EnvelopeBoundaryGuard before being acted on.
  • LocalModelBaseline/PrivateHoldout never leave the local process — never published to MQTT, never sent to the orchestrator/aggregator/Fabric.
  • A hash mismatch on the downloaded candidate model is always REJECT (fail-closed), never DEFER.
  • FederatedDockerClientExecutor/Docker path must not be modified by this module — coexistence only.
  • ReferenceMetricHoldoutEvaluator must not be presented as production-ready — it does not run real inference; document this explicitly wherever it is referenced.

Tests

Test count for the internal hop (EmbeddedRestSseBrokerTest — ephemeral/fixed port lifecycle, fail-fast on an occupied port, connection ceiling with Retry-After, body-size limit, a real publish/subscribe round-trip via two RestSseGateway instances, module federated-digital-twin-broker) is unaffected by the revert — the internal hop still uses EmbeddedRestSseBroker (BL-243). In federated-digital-twin-broker: broker config/lifecycle against the real EmbeddedTwinBroker→EmbeddedRestSseBroker delegation, envelope boundary verification, bridge relaying (happy path, invalid signature, tampered integrity, malformed-message isolation), the client command adapter (including failure isolation), the validator (PROMOTE/REJECT/DEFER, hash mismatch, download/evaluator failure), and LocalDigitalTwinRuntime end-to-end (real embedded broker, no Testcontainers needed since the "internal" side is this module's own broker) — orchestrator commands published on a fake single externalGateway (a MqttGateway test double) cross the real embedded broker and produce a real client update and twin validation decision on that same external gateway, reflecting the reverted single-gateway design. The BL-241-era regression test that proved traffic published on the "old" MQTT gateway never reached the internal broker is obsolete post-revert (there is only one external gateway again) — check the current test suite for whether it was removed or repurposed.

BL-243 timing note: because RestSseGateway.subscribe() is asynchronous (a dedicated connection thread per call) where the old EmbeddedTwinMqttGateway.subscribe() blocked until the Paho SUBACK, LocalDigitalTwinRuntimeTest waits for runtime.internalConnectionCount() >= 5 (note: a fully wired runtime with TwinArtifactChannel opens 7 internal SSE subscriptions total: 3 outbound in DigitalTwinBridge, 2 in DigitalTwinClientCommandAdapter, 1 in DigitalTwinValidatorAdapter, and 1 in TwinArtifactChannel) before publishing, and separately waits for hasPendingRoundConfig(...) between publishing RoundConfig and RoundFinalizationSignal.

FederatedExperimentChaincodeTest gained a TwinValidationDecisionTests nested class covering audit-event creation, bridge delegation, and the default-no-op case for legacy bridges.