Ir para o conteúdo

Component: Federated Orchestrator

Responsibility

The orchestrator controls the lifecycle of the federated experiment. It publishes the available contract, selects eligible clients, starts rounds, generates a signed RoundConfig, and records coordination events on the ledger.

BL-242: the orchestrator is also the only party that ever forwards a client's local update to the aggregator — the client never connects to the aggregator directly. See ClientUpdateRelay (BL-242) below.

BL-244: the orchestrator (in practice, the composition root that owns both RoundCoordinator and the aggregator — a *FederatedSession/Docker daemon) is now also the party that gathers all of a round's data and calls the aggregator directly, and the only party that does anything with the result — the aggregator itself became a pure computation function with no I/O. See BL-244 — Aggregation result publishing below.

Implementation

Item Value
Module modules/federated-orchestrator
Java package br.ufma.lsdi.orchestrator
Main class RoundCoordinator
Strategy factory FederatedRoundStrategyFactory
Strategy catalog & plans SelectionStrategyCatalog, SelectionStrategySpec, RoundParticipationPlan, RoundClientSelector
Client update relay (BL-242) ClientUpdateRelay
Client artifact ingestor ClientArtifactIngestor
Twin validation ingestor TwinValidationDecisionIngestor
Update collection (BL-244) MqttClientUpdateCollector — moved here from federated-aggregator
Artifact resolution (BL-244) ClientUpdateResolver — resolves ClientUpdateMessage→ClientUpdate
Result publishing (BL-244) AggregationResultPublisher — stores artifacts, registers the ledger, publishes GlobalModelPublication + GlobalModelPayload
Remote aggregator HTTP client (BL-244 redesign) RemoteAggregatorClient — POST /aggregate against a genuinely separate process/container's aggregator daemon
Round trigger for external participants (BL-195/BL-244) ExternalParticipantRoundTrigger — trigger(...) and triggerAndReturnConfig(...)
Note on RoundFlowPublisher Present in module as an unused helper class (publication still goes directly through MqttGateway inside RoundCoordinator/AggregationResultPublisher)

Inputs

  • active contract (ExperimentContract);
  • list of clients with valid acceptance (ClientConsentChaincode);
  • participation policy (ParticipationPolicy);
  • AggregationPolicy.participationStrategyId() as a strategy fallback;
  • state of previous rounds;
  • commands from the Admin API;
  • current global model version.

Outputs

  • signed RoundConfig (published via the injected MqttGateway — PahoMqttGateway/MQTT 5 for every composition root, JVM/Docker and Android alike; BL-241/BL-242 had briefly moved JVM/Docker to RestSseGateway/REST+SSE, reverted BL-259; see ../mqtt/01-envelope.md);
  • audit event on the ledger (registerTrainingRound);
  • list of clients selected for the round;
  • AggregationReport after the round;
  • (BL-242) relayed ClientUpdateMessage — the same signed envelope the client published, republished unchanged onto the aggregator-facing topic by ClientUpdateRelay.

Dependencies

  • ClientConsentChaincode;
  • FederatedExperimentChaincode;
  • MqttGateway;
  • HmacSignatureService;
  • LocalArtifactExperimentService (BL-244, only needed by ClientUpdateResolver/ AggregationResultPublisher — RoundCoordinator itself still has no artifact-service dependency);
  • RemoteAggregatorClient introduces a one-way Maven dependency in modules/federated-orchestrator/pom.xml on federated-aggregator solely to re-throw AggregationQuorumException; RoundCoordinator itself has no dependency on FedAvgAggregationService (composition roots wire both).

Client selection strategies

RoundCoordinator resolves the selection strategy with the following priority:

  1. ParticipationPolicy.selectionStrategyId() from the contract (layer 1).
  2. AggregationPolicy.participationStrategyId() as fallback (layer 2).
  3. all_accepted as the default strategy (layer 3).

All strategies implement FederatedRoundStrategy:

public interface FederatedRoundStrategy {
    String strategyId();
    RoundParticipationPlan plan(ExperimentContract contract, List<String> acceptedClients, long roundId);
}

AllAcceptedClientsStrategy

Selects all clients with valid acceptance (deduplicated and sorted numeric-then-lexicographically via RoundClientSelector; throws QUORUM_NOT_REACHED if < policy.minimumClients()). ID: all_accepted.

RandomSampledClientsStrategy

Selects a deterministic random fraction of the accepted clients. ID: random_sampled (or inline random_sampled:fraction=0.7:seed=42, parsed by FederatedRoundStrategyFactory).

  • DEFAULT_FRACTION = 0.7, DEFAULT_BASE_SEED = 42L (note SelectionStrategyCatalog metadata text currently describes fraction default as 0.5, whereas the implementation uses 0.7).
  • Selection uses roundSeed = baseSeed * 31L + roundId (the SHA-256 seed RoundCoordinator.deriveSeed(...) from experimentId|contractHash|roundId is placed into RoundConfig.hyperparameters.get("seed") for client training).

RoundRobinStrategy

Divides sorted clients into contiguous windows and rotates across rounds. ID: round_robin (or inline round_robin:window=<w>; default 0 computes ceil(n/2)).

  • Bounds windowSize to [minimumClients, n], picks (roundId - 1) % numWindows window, wraps from start if a tail window has fewer than minimumClients, and sets minimumExpectedUpdates = selected.size().

FederatedRoundStrategyFactory & SelectionStrategyCatalog

FederatedRoundStrategy strategy = FederatedRoundStrategyFactory.fromId("random_sampled:fraction=0.7:seed=42");
Also supports custom FQCNs via Class.forName(id) (public no-arg constructor). SelectionStrategyCatalog exposes metadata specs (all(), find(), isNative()).


RoundConfig — what the orchestrator publishes

RoundConfig (13 fields) is published via the configured MqttGateway wrapped in a signed MqttEnvelope<RoundConfig> on fl/experiments/{experimentId}/rounds/{roundId}/config:

Record Field Description
experimentId · contractVersion Experiment and contract version identifiers
roundId Sequential round number
globalModelVersion Global model version
phase RoundPhase (currently emitted as RoundPhase.ACTIVE)
globalModelHash · globalModelUri Hash and off-chain URI of the current global model (for FedProx/FedAdam)
hyperparameters Map with epochs, batch_size, learning_rate, mu, weight_decay, dropout, seed, selected_clients, minimum_expected_updates, expected_updates, minimum_total_examples, aggregation_strategy, max_update_norm, participation_strategy, non_responsive_policy, selection_rationale, round_timeout_seconds, max_rounds, termination_condition
deadline Timestamp of the maximum deadline for receiving updates
aggregationPolicy · privacyPolicy · dataWindow Policies and time window from the contract
algorithmSelection Identity and version (algorithmId, algorithmVersion) of the local algorithm required for the round (BL-155)

RoundCoordinator.startRound(...) and startRoundWithHyperparameters(...) receive the remote selection. It is folded into the RoundConfig's own evidenceString(), which is the signed content in the MqttEnvelope; any change to the algorithm or version invalidates the payload's integrity. Clients that do not declare exactly the received selection refuse the round before accessing data.


Round deadline resolution

The deadline is resolved with the following priority:

  1. ExperimentSchedule.roundTimeoutSeconds() from the contract.
  2. Environment variable ROUND_DEADLINE_SECONDS.
  3. Default fallback: 3600 seconds (1 hour).

nonResponsivePolicy

Controls the orchestrator's behavior when selected clients do not respond before the deadline:

Value Behavior
PROCEED_WITH_AVAILABLE After the deadline, uses the updates received (default)
REQUIRE_FULL_PARTICIPATION Waits for all selected clients or aborts the round
REQUIRE_QUORUM_OF_SELECTED Requires minimumClients responses or aborts

ClientUpdateRelay (BL-242)

The client never connects to the aggregator. Before BL-242, MqttClientUpdateCollector subscribed directly to the topic the client publishes to (fl/experiments/{expId}/rounds/{roundId}/clients/{clientId}/updates) — a direct client→aggregator pub-sub link that bypassed the orchestrator entirely. ClientUpdateRelay closes that gap: it is the only subscriber of the client-facing topic, and the only publisher of the new aggregator-facing topic (fl/experiments/{expId}/rounds/{roundId}/aggregator/clients/{clientId}/updates, MqttTopics.aggregatorClientUpdate(...)), which is in turn the only topic MqttClientUpdateCollector subscribes to.

public final class ClientUpdateRelay {
    public static ClientUpdateRelay forExperiment(String experimentId, MqttGateway gateway) { ... }
    public static ClientUpdateRelay forAllExperiments(MqttGateway gateway) { ... }
    public void start() { gateway.subscribe(topicFilter, ClientUpdateMessage.class, this::onUpdate); }
    private void onUpdate(String topic, MqttEnvelope<?> envelope) {
        // republishes the SAME envelope, unmodified — no re-signing.
        gateway.publish(MqttTopics.aggregatorClientUpdate(update.experimentId(), update.roundId(),
                update.clientId()), envelope);
    }
}

Deliberately a dumb relay — it republishes the original signed envelope byte-for-byte, the same rationale as DigitalTwinBridge.relayOutbound (BL-238): the payload is already signed by the client that produced it, and MqttClientUpdateCollector.onUpdateMessage still calls envelope.verify(...) at the final destination. The relay never re-verifies or re-signs — it only moves the message one hop closer to the aggregator. Per-message try/catch isolation mirrors PahoMqttGateway.dispatch (BL-140): a failure republishing one client's update is logged and never stops the relay or drops the next message.

Uniform wiring — every composition root owns its relay. Every *FederatedSession and every Docker daemon that constructs an MqttClientUpdateCollector also constructs and starts its own ClientUpdateRelay.forExperiment(experimentId, gateway) immediately afterward, on the same MqttGateway. This makes each process self-sufficient — it never depends on another process (like the Admin Monitor) being up to receive updates. AdminWebServer additionally runs a cross-experiment ClientUpdateRelay.forAllExperiments(mqttGateway) as one more defensive layer for the Docker-authoritative path (see 01-admin-api-monitor-web.md) — never a dependency for any of the sessions/daemons above, since duplicate relaying across multiple relays covering the same experiment is harmless: MqttClientUpdateCollector already deduplicates updates by (experimentId, contractVersion, roundId, clientId) before inserting.

Download direction (metadata) is unaffected. GlobalModelPublication (aggregator → client, URI/hash metadata only) is still published without a relay hop — but since BL-244 it is AggregationResultPublisher (not FedAvgAggregationService, which no longer touches MqttGateway at all) that publishes it, right after storing the model artifact. See below. The bytes of that same model now also travel this way, via a companion GlobalModelPayload — see "ClientArtifactIngestor — the data-sovereignty boundary" below.


ClientArtifactIngestor — the data-sovereignty boundary

The client never touches the Artifact Experiment Service directly. FederatedClientService takes a ClientArtifactChannel, not a LocalArtifactExperimentService — it publishes the local update/metrics/profile bytes as a signed ClientArtifactPayload on the client-facing artifact topic (MqttTopics.allClientArtifacts(experimentId)), the same digital-twin interface that already carries ClientUpdateMessage (see 17-local-digital-twin-broker.md). ClientArtifactIngestor is the orchestrator-side counterpart: it subscribes to that topic, verifies the envelope signature and the decoded content's hash, and writes the bytes into its own LocalArtifactExperimentService — under the exact namespace/filename convention ({experimentId}/{clientId}/round-{roundId}/local_update.bin, etc.) that ClientUpdateResolver recomputes to read them back. A malformed/tampered/wrongly-signed payload is logged and dropped, same isolation rationale as ClientUpdateRelay.

ClientArtifactIngestor.forExperiment(experimentId, gateway, artifactService, signatures).start();

Wired alongside ClientUpdateRelay, same lifecycle. Every composition root that constructs ClientUpdateRelay.forExperiment(...) also constructs and starts a ClientArtifactIngestor.forExperiment(...) on the same MqttGateway/artifactService — the client's artifact bytes must already be on disk by the time ClientUpdateResolver.resolve(...) tries to read them.

Delivery is asynchronous, so ClientUpdateResolver polls, not reads-once. Unlike the pre-BL-244 world where the client wrote the artifact synchronously before building ClientUpdateMessage (so it was always already on disk), the artifact and the update message now arrive as two independent network publications with no ordering guarantee between them. ClientUpdateResolver.resolve(...) retries (bounded, default 5s) both for "file not present yet" and for "file present but hash doesn't match yet" — the Artifact Experiment Service is never cleared between rounds, so a stale file from a previous round can legitimately sit at the same path for an instant before ClientArtifactIngestor overwrites it; only a hash mismatch that persists until the deadline is treated as real corruption/tampering (UPDATE_REJECTED).

Download direction, real bytes: AggregationResultPublisher now also publishes a GlobalModelPayload (the same modelText bytes it just wrote to global_model.bin, no extra I/O) right after GlobalModelPublication, so FederatedClientService.resolveGlobalModel can download the FedProx reference model through TwinArtifactChannel instead of reading the Artifact Experiment Service directly.


BL-244 — Aggregation result publishing

The aggregator never touches MQTT, the Artifact Experiment Service, or the ledger. Before BL-244, FedAvgAggregationService independently subscribed to the aggregator-facing MQTT topic, downloaded update bytes from the Artifact Experiment Service, and published GlobalModelPublication — decoupled from the orchestrator except through async pub/sub. The user asked for the inverse: the orchestrator gathers everything and calls the aggregator directly; the aggregator only computes and returns a result.

Three classes, all in this module, carry the responsibilities FedAvgAggregationService used to have:

// 1. Collect + resolve — MqttClientUpdateCollector (moved here from federated-aggregator) +
//    ClientUpdateResolver (new): turns the raw ClientUpdateMessage stream into resolved
//    ClientUpdate objects (vector already read from the artifact).
MqttClientUpdateCollector collector = new MqttClientUpdateCollector(
        gateway, signatures, new ClientUpdateResolver(artifactService));
List<ClientUpdate> resolved = collector.awaitAndResolveUpdates(
        experimentId, contractVersion, roundId, expectedUpdates, deadline);

// 2. Call the aggregator directly — plain, synchronous, in-process Java call.
AggregationResultPublisher resultPublisher = new AggregationResultPublisher(
        experimentChaincode, artifactService, gateway, signatures, aggregatorId);
try {
    AggregationReport report = resultPublisher.publish(
            contract, roundConfig, aggregator.aggregate(contract, roundConfig, resolved));
} catch (AggregationQuorumException quorumFailure) {
    // 3. Quorum failure — the aggregator already computed reportHash(); the orchestrator
    //    registers the ledger event itself, since the aggregator never touches the chaincode.
    resultPublisher.publishQuorumFailure(contract, roundConfig, quorumFailure.reportHash());
    throw quorumFailure;
} finally {
    collector.clearRound(experimentId, roundId);
}

This exact sequence is now duplicated (deliberately — same rationale as the other per-session duplication in this codebase, e.g. PahoMqttGateway/EmbeddedTwinMqttGateway) across all ~20 in-process composition roots: the 5 canonical *FederatedSession classes, their Advanced/Improved/Kaggle/MultiRound/Docker-executor variants, and DockerSmokeAggregatorDaemonApp (the one Docker-authoritative daemon where the aggregator and the RoundCoordinator already shared a process). AggregationResultPublisher.publish(...) stores the global model and the aggregation report via LocalArtifactExperimentService (returning the pure AggregationReport with globalModelUri/aggregationReportUri now filled in), registers registerAggregationHash/registerGlobalModelVersion on FederatedExperimentChaincode (BL-261: registerGlobalModelVersion now also carries the model's own weight vector as CSV, the round's hyperparameters, and aggregation metrics/ provenance — the on-chain GlobalModelAsset is no longer hash/URI-only), and publishes GlobalModelPublication via the injected MqttGateway — exactly the three things FedAvgAggregationService.aggregate(...) used to do internally before BL-244.

DockerAggregatorDaemonApp (exp-ergonomic-risk-001) — converted (BL-244, redesign delivered). It used to hand-roll its own AggregationStrategy calls, subscribing independently to MQTT and reporting back to the Admin Monitor via an HTTP callback — a container genuinely separate from the process hosting RoundCoordinator, but not following "orchestrator sends all data, aggregator responds directly". It is now a thin HTTP server: POST /aggregate, body = RemoteAggregationRequest (ExperimentContract + RoundConfig + already-resolved List<ClientUpdate>), response = the computed AggregationReport (200) or {"reportHash", "message"} (422, quorum not met) — a direct wrapper around FedAvgAggregationService.aggregate(...), no MQTT/Artifact-Service/ledger access inside the daemon at all. See 11-aggregator.md for the full protocol and the client side (RemoteAggregatorClient, this module).


TwinValidationDecisionIngestor (BL-259)

Subscribes to MqttTopics.allTwinValidationDecisions(experimentId), verifies envelope signature + payload integrity, substitutes Hashing.sha256("") if candidateModelHash is null/blank, and calls experimentChaincode.registerTwinValidationDecision(...) on the orchestrator's ledger bridge.


Mandatory rules & current enforcement

  • Contract & phase: callers (AdminMonitoringService) are expected to ensure ContractStatus.ACTIVE; RoundCoordinator.startRound itself currently skips an internal ACTIVE status check and always emits RoundPhase.ACTIVE.
  • Consent & quorum: only clients with ConsentStatus.isAccepted() enter the pool, and strategies enforce AggregationPolicy.minimumClients().
  • Non-responsive policy: br.ufma.lsdi.orchestrator.NonResponsivePolicy is @Deprecated (still referenced in RoundParticipationPlan / SelectionStrategySpec); all 3 native strategies set PROCEED_WITH_AVAILABLE and RoundCoordinator does not branch on REQUIRE_FULL_PARTICIPATION/REQUIRE_QUORUM_OF_SELECTED.
  • Publish RoundConfig wrapped in a signed MqttEnvelope (X.509 or HMAC) with timestamp, deadline (ExperimentSchedule → ROUND_DEADLINE_SECONDS → 3600s), and training seed derived from SHA-256 of experimentId|contractHash|roundId.
  • Record the round start on the ledger via FederatedExperimentChaincode.registerTrainingRound.
  • Every composition root that starts a ClientUpdateRelay must also start a ClientArtifactIngestor on the same MqttGateway/artifactService — it is the only writer of client-origin artifacts into the Artifact Experiment Service (BL-244 already forbids the aggregator from touching it; this extends the same boundary to the client). Must use the SignatureService that actually knows the real per-client identities (the one already passed to MqttClientUpdateCollector/ClientUpdateResolver in that same composition root) — never a throwaway/simulation-only identity registry, which would silently reject every real artifact (see 11-aggregator.md for the concrete AdminWebServer vs. DockerAggregatorDaemonApp distinction).

Current state

RoundCoordinator selects accepted clients using the strategy configured in the contract, blocks rounds without the minimum quorum, and publishes the signed configuration through whichever MqttGateway implementation it was constructed with. In production, that implementation is PahoMqttGateway (MQTT 5 with TLS/mTLS, module federated-containers) for every composition root, JVM/Docker and Android alike — RoundCoordinator itself is unaware of which concrete implementation is wired in, since all implement MqttGateway. (BL-241/BL-242 had briefly wired RestSseGateway/REST+SSE for JVM/Docker composition roots; that migration was reverted end to end by BL-259.) InMemoryMqttGateway is a test-only fixture (federated-common test-jar), never a production path. It supports the three native strategies (all_accepted, random_sampled, round_robin).

Sensor data simulation/ingestion (iot//mhub) uses the same single PahoMqttGateway instance as the round protocol again, post-revert — see 05-mqtt.md.

Tests

Covered by ArchitectureComponentTestSuite and RoundCoordinator-specific unit tests: - Client selection (all strategies); - RoundConfig with signature; - Configuration publication via the injected MqttGateway (tested with the in-memory fixture; round-trip over real transports is covered separately by federated-containers and federated-digital-twin-broker's own test suites); - Round blocking without quorum; - Deadline resolution.

ClientUpdateRelay (BL-242) is covered by ClientUpdateRelayTest: relaying preserves the original signature/senderId, per-experiment scoping ignores other experiments' updates, the cross-experiment variant relays across experiments, and an incompatible payload on a wildcard topic is ignored without throwing. A regression test in ErgonomicRiskFederatedSessionTest proves a ClientUpdateMessage published directly to the client-facing topic — bypassing the relay — never reaches the aggregator.

BL-244 additions, all in this module: MqttClientUpdateCollectorTest/ MqttClientUpdateCollectorDiagnosticsTest/AwaitUpdatesWaitTargetTest (moved verbatim from federated-aggregator, package renamed); ClientUpdateResolverTest (new — resolves a valid update, rejects a hash mismatch with UPDATE_REJECTED); AggregationResultPublisherTest (new — covers what FedAvgAggregationServiceTest's old LedgerSideEffectsTests/MqttPublicationTests used to cover: artifact storage + URI population, ledger registration on success, successive rounds incrementing the registered version, MQTT publication when a gateway/signature service is configured, silent no-op when they are not, and publishQuorumFailure registering the ledger event without storing an artifact).

BL-244 (redesign) additions: ExternalParticipantRoundTriggerTest's TriggerAndReturnConfigTests (returns the same incremented globalModelVersion as trigger(...), the returned RoundConfig matches the triggered round, dataset is still published exactly once); RemoteAggregatorClientTest (against a real embedded com.sun.net.httpserver.HttpServer fixture, no mocks — 200 decodes the AggregationReport, 422 throws AggregationQuorumException carrying reportHash, 500 throws IllegalStateException naming the status, an unreachable server throws IllegalStateException). See 11-aggregator.md for DockerAggregatorDaemonAppTest (the daemon side of the same protocol) and ErgonomicRiskFederatedSessionTest's triggerRoundForExternalParticipantsAggregatesRemotelyWhenConfigured (full round trip: real daemon HTTP handler behind a test server, two simulated external participants publishing artifact+update on the shared gateway, asserts the aggregation result reaches the ledger through the async path).