Component: Local Digital Twin Broker (ADR-002)¶
Responsibility¶
The local access interface to the digital twin, requested to let the orchestrator send and
receive commands/data through a dedicated component instead of the federated client being
reachable only via Docker/env vars. Implemented as an embedded broker (BL-238; HTTP+SSE since
BL-243, originally an MQTT microbroker) — the same architectural pattern already documented for
the external reference project (SESI Ergotech) in docs/cddl_cdal.md/docs/mqtt/MQTT_TOPICS_CDDL.md:
a local broker running inside the client's own process/container, serving as an internal bus,
bridged to the real external broker through a separate connection.
Closes the gap left open in 12-local-digital-twin.md: the
LocalModelValidator sketched there as "next implementation step" is implemented here.
BL-241 update (Phase 1 of BL-240), reverted (BL-259): the external hop of
DigitalTwinBridgehad briefly stopped being a single MQTT gateway —RoundConfig/RoundFinalizationSignal/GlobalModelPublication/ClientUpdateMessagemoved to REST+SSE while onlyTwinValidationDecisionMessagestayed on MQTT. That split was reverted end to end: the external hop is back to a single MQTTexternalGateway(in production,PahoMqttGateway) for every payload type,TwinValidationDecisionMessageincluded (it never actually left MQTT). TheDigitalTwinBridgeconstructor now takes oneexternalGatewayagain; a deprecated 5-argument overload (externalMqttGateway,externalRestSseGateway) is kept temporarily for any caller not yet updated, and simply ignores the second gateway. The internal hop (embedded broker, BL-243) is untouched by this revert — it still runs overEmbeddedRestSseBroker/RestSseGateway(modulefederated-digital-twin-broker), described in full further down. Only the external boundary reverted; see "Architecture" below for the updated diagram.BL-242 — no code change here, composes as-is.
DigitalTwinBridge.relayOutboundstill deliversClientUpdateMessageonto the external client-facing topic (MqttTopics.allClientUpdates(experimentId)) — exactly the topic the orchestrator's newClientUpdateRelaysubscribes to. The bridge has no notion of, and no dependency on, the aggregator-facing topic the relay republishes to. See 04-orchestrator.md.BL-243 — the internal hop also migrated, from MQTT to HTTP+SSE. The embedded MQTT microbroker (Moquette) and its dedicated
EmbeddedTwinMqttGateway(MQTT 3.1.1 via Paho v3) are gone.EmbeddedTwinBrokernow delegates toEmbeddedRestSseBroker(new production class, modulefederated-digital-twin-broker) — the same mechanism already used for the external hop since BL-241 — and the bridge/adapters connect to it via plainRestSseGatewayinstances instead of a Paho v3 client. The Moquette-specific limitation that originally motivated the MQTT 3.1.1 workaround (see "Finding" below, now historical) no longer applies: there is no MQTT anywhere in this module's internal hop.DigitalTwinBridge/DigitalTwinClientCommandAdapter/DigitalTwinValidatorAdapter/EnvelopeBoundaryGuardneeded zero changes — they only ever depended on theMqttGatewayinterface, never on the concrete transport.
Implementation¶
| Item | Value |
|---|---|
| Module | modules/federated-digital-twin-broker |
| Java package | br.ufma.lsdi.digitaltwin (validator classes in br.ufma.lsdi.digitaltwin.validator) |
| Embedded broker | EmbeddedTwinBroker delegates to EmbeddedRestSseBroker (module federated-digital-twin-broker, HTTP+SSE, com.sun.net.httpserver.HttpServer — BL-243; originally Moquette/MQTT 3.1.1, see finding below) |
| Composition root | LocalDigitalTwinRuntime |
Architecture¶
┌─ MQTT (external, reverted to single gateway BL-259) ────────────────────┐
│ External MQTT broker (Mosquitto), ssl://mosquitto:8883, PahoMqttGateway │
└──────────────────────────────────┬───────────────────────────────────────┘
│ roundConfig · roundFinalization · modelPublished ·
│ clientUpdate (outbound) · twinValidationDecision (outbound)
▼
┌──────────────────────────── DigitalTwinBridge ─────────────────────────┐
│ externalGateway (single MQTT gateway, all types) │
│ verify() + verifyPayloadIntegrity() at the boundary │
└────────────────────────────────┬────────────────────────────────────────┘
│ http://127.0.0.1:{port} — REST+SSE (BL-243,
│ internal hop only, NOT reverted),
│ same MqttTopics.* strings as topic/topicFilter
▼
┌──────────────────────────────────────────────────┐
│ EmbeddedTwinBroker → EmbeddedRestSseBroker (loopback) │
│ runs inside the client's process/container │
└──────────┬─────────────────┬───────────────────────┘
▼ ▼
DigitalTwinClientCommandAdapter DigitalTwinValidatorAdapter
- holds RoundConfig pending - receives GlobalModelPublication
- on finalization, calls - calls LocalModelValidator.validate()
FederatedClientService - publishes TwinValidationDecisionMessage back out
.executeRoundOnFinalizationSignal through the bridge (external MQTT 5, BL-259)
- registers via its OWN local FederatedExperimentChaincode
(in-memory only here — never the real ledger; see
TwinValidationDecisionIngestor, orchestrator side)
Everything below "internalGateway" (the embedded broker and both adapters) is unchanged in
behavior by BL-241/BL-243/BL-259 — only the transport carrying the bytes on the external hop
moved and then moved back (BL-241 split it into MQTT+REST+SSE; BL-259 reverted it to a single MQTT
gateway); the internal hop stayed HTTP+SSE since BL-243 throughout, untouched by the revert.
DigitalTwinBridge, both adapters, and EnvelopeBoundaryGuard only ever depend on the
MqttGateway interface — none of them needed a single line changed for any of these transport
changes.
Topic namespace: the embedded broker reuses the exact same MqttTopics.* strings as the
external broker — they are physically distinct endpoints, so there is no collision, and
FederatedClientService (which already builds topics via MqttTopics.clientUpdate(...)
internally) works unmodified when handed the internal gateway. Since BL-243 those strings are
used as topic=/topicFilter= query params over HTTP for the internal hop; the external hop uses
them as literal MQTT topic strings again (as it always had, apart from the reverted BL-241 window).
Reuse, no changes to federated-client: FederatedClientService already exposes
executeRoundEnvelopeAndPublish(...) and executeRoundOnFinalizationSignal(...), both accepting
an arbitrary MqttGateway to publish the resulting ClientUpdateMessage back. This module only
feeds them; federated-client itself was not touched.
Data-sovereignty update — no longer just coexistence. The note below ("coexistence, not migration") described this module's status before the fronteira de soberania de dados refactor:
FederatedClientServiceused to write/read the Artifact Experiment Service directly, bypassing this broker entirely. It no longer can — its constructor now takes aClientArtifactChannel, and every real production/research composition root (*FederatedSession,DockerErgonomicClientDaemonApp,DockerSmokeClientDaemonApp) wires a realEmbeddedTwinBroker+DigitalTwinBridgeand aTwinArtifactChannelbuilt on it. The one remaining exception is the ephemeral, host-orchestrated Docker path (Docker*ClientApp, e.g.DockerHarClientApp) — it has no live orchestrator/broker counterpart reachable from inside a single short-lived container, so it usesLocalDiskArtifactChannelinstead (modulefederated-client, see its Javadoc). See "ClientArtifactPayload / GlobalModelPayload / TwinArtifactChannel" below.
Coexistence, not migration (historical — see the update box above): FederatedDockerClientExecutor
and the Docker/env-vars path are untouched. This module was, at the time, an additional path.
Historical finding: Moquette 0.17 (Maven Central) only spoke MQTT 3.1.1¶
Superseded by BL-243 — kept here only as the historical rationale for why the internal hop briefly needed a second MQTT client stack, now removed entirely.
Moquette added MQTT5 support in release 0.18 — but 0.18 was never published to Maven
Central (only 0.15/0.16/0.17 are there). PahoMqttGateway, the project's real
MqttGateway implementation, is MQTT5-only (org.eclipse.paho.mqttv5.client) and Moquette 0.17
rejected its CONNECT packet ("MQTT protocol version is not valid").
Consequence at the time: the internal hop (bridge/adapters ↔ embedded broker) used a
dedicated EmbeddedTwinMqttGateway, built on the MQTT 3.1.1 client (org.eclipse.paho.client.mqttv3,
same 1.2.5 release line as the v5 client used elsewhere). BL-243 removed both the Moquette broker
and EmbeddedTwinMqttGateway — the internal hop is HTTP+SSE now, so this MQTT-version
incompatibility no longer applies to any part of this module.
Main classes¶
EmbeddedTwinBroker¶
BL-243: thin wrapper delegating to EmbeddedRestSseBroker (module federated-digital-twin-broker) — no
longer encapsulates a Moquette Server. Always binds to 127.0.0.1 (never 0.0.0.0) — the
broker is strictly local to the client's own process/container, never exposed externally.
Port resolution: EmbeddedRestSseBroker reserves the port with a plain java.net.ServerSocket
bind/close first, same technique the original Moquette-backed class used and for the same reason
— empirically, neither com.sun.net.httpserver.HttpServer.create(...) nor .start() reliably
throw when the requested port is already occupied (confirmed by direct testing before writing
this class), so a synchronous preflight probe is still required, not a Moquette-specific quirk.
This also resolves ephemeral ports (port=0) for tests. Small TOCTOU window between the
reservation and the real bind; acceptable in tests (re-runnable), production should use a fixed
port via DIGITAL_TWIN_BROKER_PORT.
EmbeddedTwinMqttGateway (the old MQTT 3.1.1 Paho-based internal gateway) is removed — the
bridge/adapters connect via plain RestSseGateway (module federated-digital-twin-broker) instances
instead, the same class already in production for the external hop since BL-241. No
TLS/authentication on the internal broker: loopback-only.
EnvelopeBoundaryGuard / BoundaryVerdict¶
Single point of envelope.verify(signatureService) + envelope.verifyPayloadIntegrity() at the
external→internal boundary (per .agents/shared-rules.md). Reused by DigitalTwinBridge,
DigitalTwinClientCommandAdapter, and DigitalTwinValidatorAdapter — every receiver re-checks,
defense-in-depth, rather than trusting an upstream check.
DigitalTwinBridge¶
Relays external↔internal in both directions: RoundConfig/RoundFinalizationSignal/
GlobalModelPublication inbound (validated); ClientUpdateMessage/TwinValidationDecisionMessage
outbound (already signed by their producer, no re-verification needed). A rejected/malformed
envelope is logged and dropped — never propagates and never blocks subsequent messages.
Single external gateway again (BL-241 dual-gateway split reverted, BL-259): the constructor
takes one externalGateway (in production, PahoMqttGateway). start() wires all inbound relays
(RoundConfig/RoundFinalizationSignal/GlobalModelPublication/GlobalModelPayload) and all
outbound relays (ClientUpdateMessage/TwinValidationDecisionMessage/ClientArtifactPayload) to
that same gateway. relayInbound/relayOutbound read the fixed externalGateway field again
(no gateway parameter). A deprecated 5-argument constructor overload
(externalMqttGateway, externalRestSseGateway) is kept temporarily for any not-yet-updated
caller — it discards the second gateway and delegates to the single-gateway constructor using the
first.
DigitalTwinClientCommandAdapter¶
Holds a RoundConfig pending only after it passes EnvelopeBoundaryGuard (executeRoundOnFinalizationSignal
does not re-verify the RoundConfig's own signature — only the finalization signal's — so this
check is this adapter's responsibility, not redundant). On a matching RoundFinalizationSignal,
removes the pending config and calls FederatedClientService.executeRoundOnFinalizationSignal(...);
failures are caught and logged, never propagated from the MQTT callback.
validator.LocalModelValidator / BaselineComparisonModelValidator¶
See 12-local-digital-twin.md
for the full step-by-step. TwinValidationOutcome (decision + evaluated metric + candidate hash +
reason) is a deliberate superset of the original sketch's bare ValidationDecision return — the
adapter needs those details to build TwinValidationDecisionMessage and the audit trail.
LocalModelBaseline/PrivateHoldout/HoldoutEvaluator/LocalModelValidator never travel over
the wire at all (by the same invariant as the rest of the digital twin — validation holdout stays
local), so they live in this module, not in federated-common. ValidationDecision and
TwinValidationDecisionMessage do travel as an MqttEnvelope payload (external MQTT 5 after
BL-259 revert, and internal HTTP+SSE loopback), so they live in federated-common alongside every other payload type.
DigitalTwinValidatorAdapter¶
Subscribes to model publications on the internal broker, runs LocalModelValidator, signs and
publishes TwinValidationDecisionMessage back out through the internal gateway (from where
DigitalTwinBridge relays it externally via MQTT 5), and calls
FederatedExperimentChaincode.registerTwinValidationDecision(...) on its own, unbridged
chaincode instance — an in-memory-only local audit trail, never the real ledger (mirrors the same
invariant established for registerLocalUpdateHash/consent in BL-257: the client/twin never
writes to Fabric directly). The authentic registration happens on the orchestrator side, once
TwinValidationDecisionIngestor receives the relayed message (BL-259, see below).
LocalDigitalTwinRuntime¶
Composition root — starts the embedded broker, opens one RestSseGateway per component
(bridge/client-adapter/validator-adapter/artifact channel, each with a distinct client ID
suffix — no longer a correctness requirement since REST+SSE connections don't collide by client ID
the way MQTT sessions did in BL-115, kept only for log/debugging clarity), wires everything, and
closes it all in reverse order. Exposes a package-private internalConnectionCount() for test
synchronization: RestSseGateway.subscribe() is asynchronous (dedicated connection thread per
call), unlike the old EmbeddedTwinMqttGateway.subscribe() (blocking, synchronous Paho SUBACK) —
a caller that publishes immediately after construction can race the internal SSE connections
still coming up.
Constructor takes a client factory, not a ready client. FederatedClientService needs a
TwinArtifactChannel built on the broker's own internal gateway — which only exists once the
broker has already started inside this constructor — so LocalDigitalTwinRuntime takes a
Function<MqttGateway, FederatedClientService> clientServiceFactory instead of a pre-built
FederatedClientService. It opens the artifact channel's internal connection first, then calls
the factory with it, then wires the client command adapter with the result.
ClientArtifactPayload / GlobalModelPayload / TwinArtifactChannel¶
Fronteira de soberania de dados: two new MqttEnvelope payload types (module federated-common)
carry the actual artifact bytes (Base64) over this same bridge — ClientArtifactPayload outbound
(client → orchestrator: local update/metrics/profile) and GlobalModelPayload inbound
(orchestrator → client: the FedProx reference model), both added to DigitalTwinBridge.start()
with one relayOutbound/relayInbound line each, no other change to the bridge.
TwinArtifactChannel (module federated-client) is the production ClientArtifactChannel
implementation: upload() computes the hash/length locally (pure function, no disk access) and
publishes a signed ClientArtifactPayload on the client-facing artifact topic;
downloadGlobalModel() subscribes, at construction, to the global-model-payload topic and serves
from an in-memory cache, waiting up to a bounded timeout (default 5s) for the payload to arrive —
a hash mismatch on an already-received payload is a hard UPDATE_REJECTED (integrity violated),
distinct from "not arrived yet" (Optional.empty(), tolerated as pure FedAvg). On the orchestrator
side, ClientArtifactIngestor (module federated-orchestrator) subscribes to the same
client-facing artifact topic and writes the decoded, hash-verified bytes into its own Artifact
Experiment Service, under the same namespace/filename convention ClientUpdateResolver uses to
read them back — the client never learns, or needs to know, the real storage path.
Body-size limit raised for real payloads (256 KiB → 8 MiB) & configuration. EmbeddedRestSseBroker defaults to DEFAULT_MAX_BODY_BYTES = 8 MiB, DEFAULT_MAX_CONNECTIONS = 16, a 15s SSE : heartbeat, and also answers 200 on catch-all / so RestSseGateway's eager constructor probe succeeds. Configuration via EmbeddedTwinBrokerConfig.fromEnvironment() (DIGITAL_TWIN_BROKER_PORT default 0, DIGITAL_TWIN_BROKER_HOST default 127.0.0.1 — note bindHost checks non-blank rather than restricting non-loopback in code) and RestSseGatewayConfig.fromEnvironment() (REST_SSE_GATEWAY_URL, REST_SSE_CLIENT_ID, REST_SSE_CONNECT_TIMEOUT_MS); RestSseGateway retries failed SSE connections with exponential backoff (500ms → 15s). (Note: BaselineComparisonModelValidator still accesses LocalArtifactExperimentService directly for model.bin and writes candidate-v{version}.bin, rather than reading GlobalModelPayload in memory.)
Ledger — registerTwinValidationDecision¶
Added to FederatedExperimentChaincode (in-memory audit event, mirrors the pattern of
registerPreflightHash/registerLocalUpdateHash) and, as default no-op methods, to both
FabricExperimentBridge and LedgerPort for backward compatibility with existing implementers
that don't need this method (e.g. TestLedgerPort).
BL-259: wired to a real Fabric transaction. FederatedExperimentSmartContract (the chaincode
in deploy/fabric/chaincode/federated-governance-java) gained a registerTwinValidationDecision
transaction, mirroring registerLocalUpdateHash's shape (appendAudit with the decision as
metadata). FabricCliLedgerMirror/FabricGatewayLedgerMirror no longer override this method as a
no-op — they invoke the transaction for real (pseudonymizing clientId first, same as every other
per-client ledger write). New TwinValidationDecisionIngestor (module federated-orchestrator,
package br.ufma.lsdi.orchestrator) is the actual subscriber: it receives the message over the
FederatedExperimentChaincode.registerTwinValidationDecision(...) on its own chaincode
instance, the same one AggregationResultPublisher already uses for registerAggregationHash,
bridged to the real network when the process is constructed with one (ledgerMode=BLOCKCHAIN).
(TwinValidationDecisionIngestor consumes the decision over the external MQTT 5 gateway.)
Still not validated end-to-end against a running Fabric network in this environment —
installing this new chaincode version requires the peer CLI and a Java chaincode builder
available on the test-network, neither present here (same limitation as BL-235). The chaincode
source itself compiles and its own Gradle test suite passes; TwinValidationDecisionIngestorTest
(4 cases, InMemoryMqttGateway + the real FederatedExperimentChaincode, unbridged) proves the
Java-side wiring is correct up to the chaincode boundary.
Wired so far only in ErgonomicRiskFederatedSession (the same pilot session BL-234's X.509
migration used first) — propagating to the other 4 canonical *FederatedSession classes and the
Docker daemon apps is a registered follow-up, not silently incomplete.
Mandatory rules¶
- The embedded broker's
bindHostmust stay loopback-only (127.0.0.1by default — noteEmbeddedTwinBrokerConfigvalidates non-blank only, so operators must avoid setting0.0.0.0or routable addresses). - Every envelope crossing external→internal must pass
EnvelopeBoundaryGuardbefore being acted on. LocalModelBaseline/PrivateHoldoutnever leave the local process — never published to MQTT, never sent to the orchestrator/aggregator/Fabric.- A hash mismatch on the downloaded candidate model is always
REJECT(fail-closed), neverDEFER. FederatedDockerClientExecutor/Docker path must not be modified by this module — coexistence only.ReferenceMetricHoldoutEvaluatormust not be presented as production-ready — it does not run real inference; document this explicitly wherever it is referenced.
Tests¶
Test count for the internal hop (EmbeddedRestSseBrokerTest — ephemeral/fixed port lifecycle,
fail-fast on an occupied port, connection ceiling with Retry-After, body-size limit, a real
publish/subscribe round-trip via two RestSseGateway instances, module
federated-digital-twin-broker) is unaffected by the revert — the internal hop still uses
EmbeddedRestSseBroker (BL-243). In federated-digital-twin-broker: broker config/lifecycle
against the real EmbeddedTwinBroker→EmbeddedRestSseBroker delegation, envelope boundary
verification, bridge relaying (happy path, invalid signature, tampered integrity,
malformed-message isolation), the client command adapter (including failure isolation), the
validator (PROMOTE/REJECT/DEFER, hash mismatch, download/evaluator failure), and
LocalDigitalTwinRuntime end-to-end (real embedded broker, no Testcontainers needed since the
"internal" side is this module's own broker) — orchestrator commands published on a fake single
externalGateway (a MqttGateway test double) cross the real embedded broker and produce a real
client update and twin validation decision on that same external gateway, reflecting the reverted
single-gateway design. The BL-241-era regression test that proved traffic published on the "old"
MQTT gateway never reached the internal broker is obsolete post-revert (there is only one external
gateway again) — check the current test suite for whether it was removed or repurposed.
BL-243 timing note: because RestSseGateway.subscribe() is asynchronous (a dedicated
connection thread per call) where the old EmbeddedTwinMqttGateway.subscribe() blocked until the
Paho SUBACK, LocalDigitalTwinRuntimeTest waits for runtime.internalConnectionCount() >= 5
(note: a fully wired runtime with TwinArtifactChannel opens 7 internal SSE subscriptions total:
3 outbound in DigitalTwinBridge, 2 in DigitalTwinClientCommandAdapter, 1 in
DigitalTwinValidatorAdapter, and 1 in TwinArtifactChannel) before publishing, and separately waits for hasPendingRoundConfig(...) between publishing RoundConfig and RoundFinalizationSignal.
FederatedExperimentChaincodeTest gained a TwinValidationDecisionTests nested class covering
audit-event creation, bridge delegation, and the default-no-op case for legacy bridges.