Component: Federated Orchestrator¶
Responsibility¶
The orchestrator controls the lifecycle of the federated experiment. It publishes the available
contract, selects eligible clients, starts rounds, generates a signed RoundConfig, and records
coordination events on the ledger.
BL-242: the orchestrator is also the only party that ever forwards a client's local update to
the aggregator — the client never connects to the aggregator directly. See
ClientUpdateRelay (BL-242) below.
BL-244: the orchestrator (in practice, the composition root that owns both RoundCoordinator
and the aggregator — a *FederatedSession/Docker daemon) is now also the party that gathers all
of a round's data and calls the aggregator directly, and the only party that does anything with
the result — the aggregator itself became a pure computation function with no I/O. See
BL-244 — Aggregation result publishing below.
Implementation¶
| Item | Value |
|---|---|
| Module | modules/federated-orchestrator |
| Java package | br.ufma.lsdi.orchestrator |
| Main class | RoundCoordinator |
| Strategy factory | FederatedRoundStrategyFactory |
| Strategy catalog & plans | SelectionStrategyCatalog, SelectionStrategySpec, RoundParticipationPlan, RoundClientSelector |
| Client update relay (BL-242) | ClientUpdateRelay |
| Client artifact ingestor | ClientArtifactIngestor |
| Twin validation ingestor | TwinValidationDecisionIngestor |
| Update collection (BL-244) | MqttClientUpdateCollector — moved here from federated-aggregator |
| Artifact resolution (BL-244) | ClientUpdateResolver — resolves ClientUpdateMessage→ClientUpdate |
| Result publishing (BL-244) | AggregationResultPublisher — stores artifacts, registers the ledger, publishes GlobalModelPublication + GlobalModelPayload |
| Remote aggregator HTTP client (BL-244 redesign) | RemoteAggregatorClient — POST /aggregate against a genuinely separate process/container's aggregator daemon |
| Round trigger for external participants (BL-195/BL-244) | ExternalParticipantRoundTrigger — trigger(...) and triggerAndReturnConfig(...) |
Note on RoundFlowPublisher |
Present in module as an unused helper class (publication still goes directly through MqttGateway inside RoundCoordinator/AggregationResultPublisher) |
Inputs¶
- active contract (
ExperimentContract); - list of clients with valid acceptance (
ClientConsentChaincode); - participation policy (
ParticipationPolicy); AggregationPolicy.participationStrategyId()as a strategy fallback;- state of previous rounds;
- commands from the Admin API;
- current global model version.
Outputs¶
- signed
RoundConfig(published via the injectedMqttGateway—PahoMqttGateway/MQTT 5 for every composition root, JVM/Docker and Android alike; BL-241/BL-242 had briefly moved JVM/Docker toRestSseGateway/REST+SSE, reverted BL-259; see ../mqtt/01-envelope.md); - audit event on the ledger (
registerTrainingRound); - list of clients selected for the round;
AggregationReportafter the round;- (BL-242) relayed
ClientUpdateMessage— the same signed envelope the client published, republished unchanged onto the aggregator-facing topic byClientUpdateRelay.
Dependencies¶
ClientConsentChaincode;FederatedExperimentChaincode;MqttGateway;HmacSignatureService;LocalArtifactExperimentService(BL-244, only needed byClientUpdateResolver/AggregationResultPublisher—RoundCoordinatoritself still has no artifact-service dependency);RemoteAggregatorClientintroduces a one-way Maven dependency inmodules/federated-orchestrator/pom.xmlonfederated-aggregatorsolely to re-throwAggregationQuorumException;RoundCoordinatoritself has no dependency onFedAvgAggregationService(composition roots wire both).
Client selection strategies¶
RoundCoordinator resolves the selection strategy with the following priority:
ParticipationPolicy.selectionStrategyId()from the contract (layer 1).AggregationPolicy.participationStrategyId()as fallback (layer 2).all_acceptedas the default strategy (layer 3).
All strategies implement FederatedRoundStrategy:
public interface FederatedRoundStrategy {
String strategyId();
RoundParticipationPlan plan(ExperimentContract contract, List<String> acceptedClients, long roundId);
}
AllAcceptedClientsStrategy¶
Selects all clients with valid acceptance (deduplicated and sorted numeric-then-lexicographically via RoundClientSelector; throws QUORUM_NOT_REACHED if < policy.minimumClients()). ID: all_accepted.
RandomSampledClientsStrategy¶
Selects a deterministic random fraction of the accepted clients. ID: random_sampled (or inline random_sampled:fraction=0.7:seed=42, parsed by FederatedRoundStrategyFactory).
DEFAULT_FRACTION = 0.7,DEFAULT_BASE_SEED = 42L(noteSelectionStrategyCatalogmetadata text currently describesfractiondefault as0.5, whereas the implementation uses0.7).- Selection uses
roundSeed = baseSeed * 31L + roundId(the SHA-256 seedRoundCoordinator.deriveSeed(...)fromexperimentId|contractHash|roundIdis placed intoRoundConfig.hyperparameters.get("seed")for client training).
RoundRobinStrategy¶
Divides sorted clients into contiguous windows and rotates across rounds. ID: round_robin (or inline round_robin:window=<w>; default 0 computes ceil(n/2)).
- Bounds
windowSizeto[minimumClients, n], picks(roundId - 1) % numWindowswindow, wraps from start if a tail window has fewer thanminimumClients, and setsminimumExpectedUpdates = selected.size().
FederatedRoundStrategyFactory & SelectionStrategyCatalog¶
FederatedRoundStrategy strategy = FederatedRoundStrategyFactory.fromId("random_sampled:fraction=0.7:seed=42");
Class.forName(id) (public no-arg constructor). SelectionStrategyCatalog exposes metadata specs (all(), find(), isNative()).
RoundConfig — what the orchestrator publishes¶
RoundConfig (13 fields) is published via the configured MqttGateway wrapped in a signed MqttEnvelope<RoundConfig> on fl/experiments/{experimentId}/rounds/{roundId}/config:
| Record Field | Description |
|---|---|
experimentId · contractVersion |
Experiment and contract version identifiers |
roundId |
Sequential round number |
globalModelVersion |
Global model version |
phase |
RoundPhase (currently emitted as RoundPhase.ACTIVE) |
globalModelHash · globalModelUri |
Hash and off-chain URI of the current global model (for FedProx/FedAdam) |
hyperparameters |
Map with epochs, batch_size, learning_rate, mu, weight_decay, dropout, seed, selected_clients, minimum_expected_updates, expected_updates, minimum_total_examples, aggregation_strategy, max_update_norm, participation_strategy, non_responsive_policy, selection_rationale, round_timeout_seconds, max_rounds, termination_condition |
deadline |
Timestamp of the maximum deadline for receiving updates |
aggregationPolicy · privacyPolicy · dataWindow |
Policies and time window from the contract |
algorithmSelection |
Identity and version (algorithmId, algorithmVersion) of the local algorithm required for the round (BL-155) |
RoundCoordinator.startRound(...) and startRoundWithHyperparameters(...) receive the remote
selection. It is folded into the RoundConfig's own evidenceString(), which is the signed
content in the MqttEnvelope; any change to the algorithm or version invalidates the payload's
integrity. Clients that do not declare exactly the received selection refuse the round before
accessing data.
Round deadline resolution¶
The deadline is resolved with the following priority:
ExperimentSchedule.roundTimeoutSeconds()from the contract.- Environment variable
ROUND_DEADLINE_SECONDS. - Default fallback:
3600seconds (1 hour).
nonResponsivePolicy¶
Controls the orchestrator's behavior when selected clients do not respond before the deadline:
| Value | Behavior |
|---|---|
PROCEED_WITH_AVAILABLE |
After the deadline, uses the updates received (default) |
REQUIRE_FULL_PARTICIPATION |
Waits for all selected clients or aborts the round |
REQUIRE_QUORUM_OF_SELECTED |
Requires minimumClients responses or aborts |
ClientUpdateRelay (BL-242)¶
The client never connects to the aggregator. Before BL-242, MqttClientUpdateCollector
subscribed directly to the topic the client publishes to
(fl/experiments/{expId}/rounds/{roundId}/clients/{clientId}/updates) — a direct client→aggregator
pub-sub link that bypassed the orchestrator entirely. ClientUpdateRelay closes that gap: it is
the only subscriber of the client-facing topic, and the only publisher of the new
aggregator-facing topic (fl/experiments/{expId}/rounds/{roundId}/aggregator/clients/{clientId}/updates,
MqttTopics.aggregatorClientUpdate(...)), which is in turn the only topic
MqttClientUpdateCollector subscribes to.
public final class ClientUpdateRelay {
public static ClientUpdateRelay forExperiment(String experimentId, MqttGateway gateway) { ... }
public static ClientUpdateRelay forAllExperiments(MqttGateway gateway) { ... }
public void start() { gateway.subscribe(topicFilter, ClientUpdateMessage.class, this::onUpdate); }
private void onUpdate(String topic, MqttEnvelope<?> envelope) {
// republishes the SAME envelope, unmodified — no re-signing.
gateway.publish(MqttTopics.aggregatorClientUpdate(update.experimentId(), update.roundId(),
update.clientId()), envelope);
}
}
Deliberately a dumb relay — it republishes the original signed envelope byte-for-byte, the
same rationale as DigitalTwinBridge.relayOutbound (BL-238): the payload is already signed by the
client that produced it, and MqttClientUpdateCollector.onUpdateMessage still calls
envelope.verify(...) at the final destination. The relay never re-verifies or re-signs — it only
moves the message one hop closer to the aggregator. Per-message try/catch isolation mirrors
PahoMqttGateway.dispatch (BL-140): a failure republishing one client's update is logged and
never stops the relay or drops the next message.
Uniform wiring — every composition root owns its relay. Every *FederatedSession and every
Docker daemon that constructs an MqttClientUpdateCollector also constructs and starts its own
ClientUpdateRelay.forExperiment(experimentId, gateway) immediately afterward, on the same
MqttGateway. This makes each process self-sufficient — it never depends on another process (like
the Admin Monitor) being up to receive updates. AdminWebServer additionally runs a cross-experiment
ClientUpdateRelay.forAllExperiments(mqttGateway) as one more defensive layer for the
Docker-authoritative path (see 01-admin-api-monitor-web.md) — never
a dependency for any of the sessions/daemons above, since duplicate relaying across multiple
relays covering the same experiment is harmless: MqttClientUpdateCollector already deduplicates
updates by (experimentId, contractVersion, roundId, clientId) before inserting.
Download direction (metadata) is unaffected. GlobalModelPublication (aggregator → client,
URI/hash metadata only) is still published without a relay hop — but since BL-244 it is
AggregationResultPublisher (not FedAvgAggregationService, which no longer touches MqttGateway
at all) that publishes it, right after storing the model artifact. See below. The bytes of that
same model now also travel this way, via a companion GlobalModelPayload — see
"ClientArtifactIngestor — the data-sovereignty boundary" below.
ClientArtifactIngestor — the data-sovereignty boundary¶
The client never touches the Artifact Experiment Service directly. FederatedClientService
takes a ClientArtifactChannel, not a LocalArtifactExperimentService — it publishes the local
update/metrics/profile bytes as a signed ClientArtifactPayload on the client-facing artifact
topic (MqttTopics.allClientArtifacts(experimentId)), the same digital-twin interface that already
carries ClientUpdateMessage (see
17-local-digital-twin-broker.md).
ClientArtifactIngestor is the orchestrator-side counterpart: it subscribes to that topic,
verifies the envelope signature and the decoded content's hash, and writes the bytes into its
own LocalArtifactExperimentService — under the exact namespace/filename convention
({experimentId}/{clientId}/round-{roundId}/local_update.bin, etc.) that ClientUpdateResolver
recomputes to read them back. A malformed/tampered/wrongly-signed payload is logged and dropped,
same isolation rationale as ClientUpdateRelay.
Wired alongside ClientUpdateRelay, same lifecycle. Every composition root that constructs
ClientUpdateRelay.forExperiment(...) also constructs and starts a
ClientArtifactIngestor.forExperiment(...) on the same MqttGateway/artifactService — the
client's artifact bytes must already be on disk by the time ClientUpdateResolver.resolve(...)
tries to read them.
Delivery is asynchronous, so ClientUpdateResolver polls, not reads-once. Unlike the pre-BL-244
world where the client wrote the artifact synchronously before building ClientUpdateMessage (so
it was always already on disk), the artifact and the update message now arrive as two independent
network publications with no ordering guarantee between them. ClientUpdateResolver.resolve(...)
retries (bounded, default 5s) both for "file not present yet" and for "file present but hash
doesn't match yet" — the Artifact Experiment Service is never cleared between rounds, so a stale
file from a previous round can legitimately sit at the same path for an instant before
ClientArtifactIngestor overwrites it; only a hash mismatch that persists until the deadline is
treated as real corruption/tampering (UPDATE_REJECTED).
Download direction, real bytes: AggregationResultPublisher now also publishes a
GlobalModelPayload (the same modelText bytes it just wrote to global_model.bin, no extra I/O)
right after GlobalModelPublication, so FederatedClientService.resolveGlobalModel can download
the FedProx reference model through TwinArtifactChannel instead of reading the Artifact
Experiment Service directly.
BL-244 — Aggregation result publishing¶
The aggregator never touches MQTT, the Artifact Experiment Service, or the ledger. Before
BL-244, FedAvgAggregationService independently subscribed to the aggregator-facing MQTT topic,
downloaded update bytes from the Artifact Experiment Service, and published
GlobalModelPublication — decoupled from the orchestrator except through async pub/sub. The user
asked for the inverse: the orchestrator gathers everything and calls the aggregator directly; the
aggregator only computes and returns a result.
Three classes, all in this module, carry the responsibilities FedAvgAggregationService used to
have:
// 1. Collect + resolve — MqttClientUpdateCollector (moved here from federated-aggregator) +
// ClientUpdateResolver (new): turns the raw ClientUpdateMessage stream into resolved
// ClientUpdate objects (vector already read from the artifact).
MqttClientUpdateCollector collector = new MqttClientUpdateCollector(
gateway, signatures, new ClientUpdateResolver(artifactService));
List<ClientUpdate> resolved = collector.awaitAndResolveUpdates(
experimentId, contractVersion, roundId, expectedUpdates, deadline);
// 2. Call the aggregator directly — plain, synchronous, in-process Java call.
AggregationResultPublisher resultPublisher = new AggregationResultPublisher(
experimentChaincode, artifactService, gateway, signatures, aggregatorId);
try {
AggregationReport report = resultPublisher.publish(
contract, roundConfig, aggregator.aggregate(contract, roundConfig, resolved));
} catch (AggregationQuorumException quorumFailure) {
// 3. Quorum failure — the aggregator already computed reportHash(); the orchestrator
// registers the ledger event itself, since the aggregator never touches the chaincode.
resultPublisher.publishQuorumFailure(contract, roundConfig, quorumFailure.reportHash());
throw quorumFailure;
} finally {
collector.clearRound(experimentId, roundId);
}
This exact sequence is now duplicated (deliberately — same rationale as the other per-session
duplication in this codebase, e.g. PahoMqttGateway/EmbeddedTwinMqttGateway) across all
~20 in-process composition roots: the 5 canonical *FederatedSession classes, their
Advanced/Improved/Kaggle/MultiRound/Docker-executor variants, and
DockerSmokeAggregatorDaemonApp (the one Docker-authoritative daemon where the aggregator and
the RoundCoordinator already shared a process). AggregationResultPublisher.publish(...)
stores the global model and the aggregation report via LocalArtifactExperimentService
(returning the pure AggregationReport with globalModelUri/aggregationReportUri now
filled in), registers registerAggregationHash/registerGlobalModelVersion on
FederatedExperimentChaincode (BL-261: registerGlobalModelVersion now also carries the
model's own weight vector as CSV, the round's hyperparameters, and aggregation metrics/
provenance — the on-chain GlobalModelAsset is no longer hash/URI-only), and publishes
GlobalModelPublication via the injected
MqttGateway — exactly the three things FedAvgAggregationService.aggregate(...) used to do
internally before BL-244.
DockerAggregatorDaemonApp (exp-ergonomic-risk-001) — converted (BL-244, redesign delivered).
It used to hand-roll its own AggregationStrategy calls, subscribing independently to MQTT and
reporting back to the Admin Monitor via an HTTP callback — a container genuinely separate from
the process hosting RoundCoordinator, but not following "orchestrator sends all data,
aggregator responds directly". It is now a thin HTTP server: POST /aggregate, body =
RemoteAggregationRequest (ExperimentContract + RoundConfig + already-resolved
List<ClientUpdate>), response = the computed AggregationReport (200) or {"reportHash",
"message"} (422, quorum not met) — a direct wrapper around
FedAvgAggregationService.aggregate(...), no MQTT/Artifact-Service/ledger access inside
the daemon at all. See 11-aggregator.md
for the full protocol and the client side (RemoteAggregatorClient, this module).
TwinValidationDecisionIngestor (BL-259)¶
Subscribes to MqttTopics.allTwinValidationDecisions(experimentId), verifies envelope signature + payload integrity, substitutes Hashing.sha256("") if candidateModelHash is null/blank, and calls experimentChaincode.registerTwinValidationDecision(...) on the orchestrator's ledger bridge.
Mandatory rules & current enforcement¶
- Contract & phase: callers (
AdminMonitoringService) are expected to ensureContractStatus.ACTIVE;RoundCoordinator.startRounditself currently skips an internalACTIVEstatus check and always emitsRoundPhase.ACTIVE. - Consent & quorum: only clients with
ConsentStatus.isAccepted()enter the pool, and strategies enforceAggregationPolicy.minimumClients(). - Non-responsive policy:
br.ufma.lsdi.orchestrator.NonResponsivePolicyis@Deprecated(still referenced inRoundParticipationPlan/SelectionStrategySpec); all 3 native strategies setPROCEED_WITH_AVAILABLEandRoundCoordinatordoes not branch onREQUIRE_FULL_PARTICIPATION/REQUIRE_QUORUM_OF_SELECTED. - Publish
RoundConfigwrapped in a signedMqttEnvelope(X.509 or HMAC) with timestamp, deadline (ExperimentSchedule→ROUND_DEADLINE_SECONDS→3600s), and training seed derived from SHA-256 ofexperimentId|contractHash|roundId. - Record the round start on the ledger via
FederatedExperimentChaincode.registerTrainingRound. - Every composition root that starts a
ClientUpdateRelaymust also start aClientArtifactIngestoron the sameMqttGateway/artifactService— it is the only writer of client-origin artifacts into the Artifact Experiment Service (BL-244 already forbids the aggregator from touching it; this extends the same boundary to the client). Must use theSignatureServicethat actually knows the real per-client identities (the one already passed toMqttClientUpdateCollector/ClientUpdateResolverin that same composition root) — never a throwaway/simulation-only identity registry, which would silently reject every real artifact (see 11-aggregator.md for the concreteAdminWebServervs.DockerAggregatorDaemonAppdistinction).
Current state¶
RoundCoordinator selects accepted clients using the strategy configured in the contract, blocks
rounds without the minimum quorum, and publishes the signed configuration through whichever
MqttGateway implementation it was constructed with. In production, that implementation is
PahoMqttGateway (MQTT 5 with TLS/mTLS, module federated-containers) for every composition root,
JVM/Docker and Android alike — RoundCoordinator itself is unaware of which concrete
implementation is wired in, since all implement MqttGateway. (BL-241/BL-242 had briefly wired
RestSseGateway/REST+SSE for JVM/Docker composition roots; that migration was reverted end to end
by BL-259.) InMemoryMqttGateway is a test-only fixture (federated-common test-jar), never a
production path. It supports the three native strategies (all_accepted, random_sampled,
round_robin).
Sensor data simulation/ingestion (iot//mhub) uses the same single PahoMqttGateway instance as
the round protocol again, post-revert — see 05-mqtt.md.
Tests¶
Covered by ArchitectureComponentTestSuite and RoundCoordinator-specific unit tests:
- Client selection (all strategies);
- RoundConfig with signature;
- Configuration publication via the injected MqttGateway (tested with the in-memory fixture;
round-trip over real transports is covered separately by federated-containers and
federated-digital-twin-broker's own test suites);
- Round blocking without quorum;
- Deadline resolution.
ClientUpdateRelay (BL-242) is covered by ClientUpdateRelayTest: relaying preserves the
original signature/senderId, per-experiment scoping ignores other experiments' updates, the
cross-experiment variant relays across experiments, and an incompatible payload on a wildcard
topic is ignored without throwing. A regression test in ErgonomicRiskFederatedSessionTest proves
a ClientUpdateMessage published directly to the client-facing topic — bypassing the relay — never
reaches the aggregator.
BL-244 additions, all in this module: MqttClientUpdateCollectorTest/
MqttClientUpdateCollectorDiagnosticsTest/AwaitUpdatesWaitTargetTest (moved verbatim from
federated-aggregator, package renamed); ClientUpdateResolverTest (new — resolves a valid
update, rejects a hash mismatch with UPDATE_REJECTED); AggregationResultPublisherTest (new —
covers what FedAvgAggregationServiceTest's old LedgerSideEffectsTests/MqttPublicationTests
used to cover: artifact storage + URI population, ledger registration on success, successive
rounds incrementing the registered version, MQTT publication when a gateway/signature service is
configured, silent no-op when they are not, and publishQuorumFailure registering the ledger
event without storing an artifact).
BL-244 (redesign) additions: ExternalParticipantRoundTriggerTest's
TriggerAndReturnConfigTests (returns the same incremented globalModelVersion as trigger(...),
the returned RoundConfig matches the triggered round, dataset is still published exactly once);
RemoteAggregatorClientTest (against a real embedded com.sun.net.httpserver.HttpServer fixture,
no mocks — 200 decodes the AggregationReport, 422 throws AggregationQuorumException carrying
reportHash, 500 throws IllegalStateException naming the status, an unreachable server throws
IllegalStateException). See 11-aggregator.md
for DockerAggregatorDaemonAppTest (the daemon side of the same protocol) and
ErgonomicRiskFederatedSessionTest's triggerRoundForExternalParticipantsAggregatesRemotelyWhenConfigured
(full round trip: real daemon HTTP handler behind a test server, two simulated external
participants publishing artifact+update on the shared gateway, asserts the aggregation result
reaches the ledger through the async path).