Ir para o conteúdo

Component: Federated Aggregator

Responsibility

The aggregator validates, selects, and combines authorized local updates into a candidate global model. Since BL-244, it is a pure computation function — it accepts a resolved List<ClientUpdate> (vectors already embedded) and an ExperimentContract/RoundConfig directly from its caller, and returns an AggregationReport. It performs zero I/O: no artifact-service access, no MqttGateway, no chaincode/ledger writes. Gathering the round's data (subscribing for client updates, resolving artifact bytes into vectors) and everything that happens with the returned report (storing the global model and report artifacts, registering ledger hashes, publishing GlobalModelPublication) are now the orchestrator's responsibility — see 04-orchestrator.md.

Why: explicit user request — "the orchestrator must send all parameters and data to the aggregator, and the aggregator must respond directly to the orchestrator, so the aggregator must not access any data from the artifact service or MQTT." Before this, FedAvgAggregationService independently subscribed to MQTT (MqttClientUpdateCollector, then in this same module), downloaded update bytes from the Artifact Experiment Service, and published results via MQTT — decoupled from the orchestrator except through async pub/sub. Now the orchestrator (a *FederatedSession/daemon composition root) calls aggregator.aggregate(...) as a plain, synchronous, in-process Java method call.

Implementation

Item Value
Module modules/federated-aggregator
Java package br.ufma.lsdi.aggregator
Main class FedAvgAggregationService — pure computation, no I/O (BL-244)
Quorum failure AggregationQuorumException (extends FederatedException, same ErrorCode.QUORUM_NOT_REACHED) — carries the pre-computed reportHash() so the caller can register the ledger failure without recomputing it
Update collection MqttClientUpdateCollector — moved to federated-orchestrator (BL-244); see 04-orchestrator.md
Artifact resolution ClientUpdateResolver — new, in federated-orchestrator (BL-244); resolves ClientUpdateMessage→ClientUpdate by reading the artifact
Result publishing AggregationResultPublisher — new, in federated-orchestrator (BL-244); stores artifacts, registers the ledger, publishes GlobalModelPublication

Inputs

  • List<ClientUpdate> — already resolved by the caller (vector embedded); the aggregator never receives a raw ClientUpdateMessage (URI+hash) nor touches MQTT/the artifact service to resolve one itself (BL-244);
  • aggregation policy (via ExperimentContract/AggregationPolicy);
  • experiment contract;
  • RoundConfig (for roundId/globalModelVersion);
  • optionally, the current global model vector (double[], for stateful strategies like FedAdam/FedYogi).

Outputs

  • AggregationReport, returned directly to the caller — globalModelUri/aggregationReportUri are always empty strings (only the caller, after storing the artifacts via AggregationResultPublisher, knows the final URI); globalModelHash/reportHash are still computed by the aggregator itself (SHA-256 of in-memory data — not I/O, and it gives the caller a free integrity check against what the artifact-service returns on write).

Mandatory rules

  • Recompute the hash of every downloaded update (done by ClientUpdateResolver, in the caller).
  • Reject an update without a valid signature.
  • Reject a duplicate update from a client in the same round.
  • Require a minimum quorum — failure throws AggregationQuorumException, without touching the ledger directly on the pure aggregate(...) path.
  • Apply a per-client weight limit.
  • Validate the update's norm, dimension, hash, signature, counts, and finite values.
  • Do not automatically promote the global model.
  • On the BL-244 path, never write to the Artifact Experiment Service, MQTT, or Fabric directly — the caller (AggregationResultPublisher, in federated-orchestrator) writes the global model and registers hashes on Fabric. (Caveat: FedAvgAggregationService still preserves its legacy experimentChaincode/artifactService/mqttGateway fields, legacy aggregateFromMessages(...) path used by HarKaggleOptimizer, and calls registerAggregationHash/registerGlobalModelVersion when constructed with a non-null chaincode; a legacy MqttClientUpdateCollector copy also still resides in this module alongside the active one in federated-orchestrator.)
  • Never send the contractual test dataset to clients.
  • Evaluate local models and the global model against the researcher-defined test dataset when EvaluationDatasetSpec is present.

Available aggregation strategies

All implement the AggregationStrategy interface with three methods: selectUpdates(List<ClientUpdate>, AggregationPolicy), computeWeights(List<ClientUpdate>, AggregationPolicy), combine(List<ClientUpdate>, Map<String,Double>, double[]).

Canonical ID Class Characteristics
fedavg FedAvgAggregationStrategyImpl Weighted average by sample count (also aliases fedavg_*, FedAvg-capped, FedAvg)
fedmedian FedMedianAggregationStrategy Coordinate-wise median (Yin et al., 2018); empty weight map, robust up to ⌊(n-1)/2⌋ Byzantine clients
trimmed_mean TrimmedMeanAggregationStrategy Trims floor(n * trimFraction) clients with lowest and highest L2 vector norm (updateNorm, default 0.1), then uniform average
trimmed_mean:trim=0.2 TrimmedMeanAggregationStrategy(0.2) Inline parameter in the ID ([0.0, 0.5))
krum KrumAggregationStrategy Selects the m updates with the lowest Euclidean distance score over n - f - 2 neighbors
krum:f=2 KrumAggregationStrategy(2, 1) f Byzantine + m=1 selected
multikrum KrumAggregationStrategy(f, m) Krum with multiple selections (f=1, m=2 default)
multikrum:f=1:m=3 KrumAggregationStrategy(1, 3) Inline parameters: f tolerated Byzantine clients + m selected updates
fedadam FedAdamAggregationStrategy Adam with momentum on global weights; stateful across rounds (falls back to weighted average if currentGlobal == null or dimension differs)
fedyogi FedAdamAggregationStrategy.fedYogi() Server-side Yogi (Reddi et al., 2021); implemented inside FedAdamAggregationStrategy (useYogi = true)

To use a strategy:

// Via contract (recommended — 6-arg or 7-arg AggregationPolicy constructor)
AggregationPolicy policy = new AggregationPolicy("fedmedian", 2, 100, 1.0, 0.0, 5000.0, "all_accepted");

// Via factory
AggregationStrategy s = AggregationStrategyFactory.fromId("trimmed_mean:trim=0.2");

// Custom strategy (full class on the classpath)
AggregationPolicy custom = new AggregationPolicy("br.ufma.MyStrategy", 2, 100, 1.0, 0.0, 5000.0, "all_accepted");

Queryable catalog:

Collection<AggregationStrategySpec> all = AggregationStrategyCatalog.all();
Optional<AggregationStrategySpec> spec  = AggregationStrategyCatalog.find("fedmedian");
boolean isNative = AggregationStrategyCatalog.isNative("br.ufma.MyStrategy"); // false

Current state

The module implements FedAvg with capped weights, client signature verification when the signature service is available, duplicate rejection, and dimension/norm/SHA-256 validation (note: rejectOutlierUpdates and median/MAD outlier filtering are not present in the current codebase). Since BL-244, the canonical server path calls FedAvgAggregationService via the (AggregationStrategy) / (AggregationStrategy, SignatureService) pure constructors and delegates artifact/MQTT/ledger I/O to AggregationResultPublisher (federated-orchestrator), while keeping the legacy I/O constructors and aggregateFromMessages(...) temporarily in place for legacy callers such as HarKaggleOptimizer (see modules/federated-aggregator/README.md). InMemoryMqttGateway remains a test-only fixture.

BL-242 (superseded in shape, not in intent, by BL-244): the aggregator-facing MQTT topic (MqttTopics.AGGREGATOR_CLIENT_UPDATE_WILDCARD, fed exclusively by the orchestrator's ClientUpdateRelay) is still how client updates physically travel from a separate client process to the composition root that now calls the aggregator directly — but the subscriber of that topic (MqttClientUpdateCollector) now lives in federated-orchestrator, not here. See 04-orchestrator.md and 04-orchestrator.md.

In the HAR experiment, the aggregator receives the full MLP weights. When the contract defines the researcher's official test dataset, the flow evaluates each submitted local model and the candidate global model against that central holdout — this evaluation step now happens in the caller, after AggregationResultPublisher.publish(...) returns the report with URIs filled in. The metrics are placed in AggregationReport as clientEvaluationMetrics, globalEvaluationMetric, evaluationMetricName, and evaluationDatasetHash.

Tests

FedAvgAggregationServiceTest (modules/federated-aggregator) — reorganized for the pure-computation shape (BL-244): QuorumTests (4, now asserting AggregationQuorumException/reportHash()), SecurityValidationTests (11), CombinationTests (1), PureComputationTests (2, new — asserts blank URIs / non-blank hashes / "candidate" promotion decision on every successful report). The ledger/MQTT side-effect coverage that used to live here (LedgerSideEffectsTests, MqttPublicationTests) moved to AggregationResultPublisherTest in federated-orchestrator.

22 unit tests in modules/federated-aggregator/src/test/java/.../AggregationStrategyTest.java (unchanged by BL-244 — the strategies were always pure):

  • FedAvg (5): weighting by sample count, weights sum to 1.0, selection without rejection, empty list throws exception
  • FedMedian (3): coordinate-wise median, robustness to an outlier, empty weight map
  • TrimmedMean (2): trim=0 equals FedAvg, trim>0 discards extremes
  • Krum (2): selects the least distant update, rejects an extreme outlier
  • Factory (7): resolution by ID, fallback to FedAvg on null/empty, exception for unknown ID
  • Catalog (3): lists native IDs, find(), isNative()

Implemented invariants

  • The aggregation report's evidence string is serialized deterministically: weights are sorted by clientId (canonicalWeights()) and rejections are sorted lexicographically (canonicalRejected()). This ensures the report's SHA-256 hash is reproducible and comparable across runs.
  • MqttClientUpdateCollector (now federated-orchestrator) exposes clearRound(experimentId, roundId) to free the memory of collected updates after aggregation completes, and awaitAndResolveUpdates(...) (BL-244) to fetch already-resolved List<ClientUpdate> in one call. The caller (the composition root) is responsible for invoking clearRound when each round ends.
  • FedAvgAggregationService.forContract(contract, ...) instantiates the strategy defined in AggregationPolicy.defaultStrategy() via AggregationStrategyFactory.fromId() — never uses a hardcoded FedAvg.
  • AggregationResultPublisher.publish(...) (BL-244, federated-orchestrator) is idempotent-adjacent to the aggregator's pure computation: given the same AggregationReport, it always stores the same artifact bytes and registers the same ledger hashes — no aggregation logic lives there, only I/O.

Docker-authoritative aggregation (separate from this module)

When Docker participants are authoritative for an experiment (BL-116), aggregation for exp-ergonomic-risk-001 (and its dynamically-deployed clones) runs in a standalone, persistent daemon process — DockerAggregatorDaemonApp — genuinely separate from the process hosting RoundCoordinator (the Admin Monitor). DockerSmokeAggregatorDaemonApp (the synthetic-smoke experiment) instead already co-locates RoundCoordinator+FedAvgAggregationService in one process/container, so it was converted straight to the in-process BL-244 pure-computation call (aggregator.aggregate(...) + resultPublisher.publish(...)), same as the 5 canonical *FederatedSession classes — the redesign below is specific to DockerAggregatorDaemonApp.

DockerAggregatorDaemonApp redesign (BL-244, delivered)

Before: the daemon self-supplied — it independently subscribed to fl/experiments/{id}/rounds/+/config over MQTT, ran its own MqttClientUpdateCollector/ ClientUpdateRelay/ClientArtifactIngestor, aggregated with a locally-tracked SequentialRoundDispatcher, and reported back to the Admin Monitor asynchronously via the AdminWebServer.handleAggregationResult HTTP callback — a design that never followed "the orchestrator sends all the data, the aggregator only responds" (unlike every in-process *FederatedSession, which does, since BL-244's first phase).

After: the daemon is now a thin, stateless HTTP server around FedAvgAggregationService.aggregate(...) — no MQTT, Artifact Experiment Service, or ledger access inside the daemon at all:

// DockerAggregatorDaemonApp — POST /aggregate, the only endpoint.
static AggregationReport aggregate(RemoteAggregationRequest request) {
    FedAvgAggregationService aggregator = FedAvgAggregationService.forContract(request.contract(), null);
    return aggregator.aggregate(request.contract(), request.roundConfig(), request.updates());
}

RemoteAggregationRequest (federated-common) is ExperimentContract + RoundConfig + already-resolved List<ClientUpdate>, encoded via JsonCodec (the same codec already used for the local AdminExperimentExecutor↔IsolatedExperimentRunner stdin/stdout protocol, BL-236) — a success response (200) is the computed AggregationReport; a quorum failure is 422 with {"reportHash":"...","message":"..."}, re-thrown by the caller as AggregationQuorumException.

Who resolves updates and calls the daemon: ErgonomicRiskFederatedSession — the same process hosting RoundCoordinator for this topology — via RemoteAggregatorClient (federated-orchestrator), see 04-orchestrator.md. Concretely, triggerRoundForExternalParticipants now calls ExternalParticipantRoundTrigger .triggerAndReturnConfig(...) (instead of .trigger(...)) to also get back the published RoundConfig, then submits a single background task that: awaits/resolves the round's updates via its own MqttClientUpdateCollector/ClientUpdateResolver (the exact same instances the in-process executeRound path uses — no new collection machinery), calls RemoteAggregatorClient.aggregate(...), and calls AggregationResultPublisher.publish(...) / publishQuorumFailure(...) on success/quorum-failure — never blocking the caller of triggerRoundForExternalParticipants itself. ClientUpdateRelay/ClientArtifactIngestor for this experiment are therefore wired only once, in the session, exactly as for every in-process composition root — the daemon needs neither.

Opt-in, off by default. The session only starts the background remote-aggregation task when the environment variable (or system property — same System.getProperty > System.getenv > default precedence as RestSseGatewayConfig) AGGREGATOR_API_URL is configured; without it, triggerRoundForExternalParticipants behaves exactly as before the redesign (publishes the dataset only) — the default in every test and in any deployment that has not yet stood up the persistent aggregator container.

Reachability — DockerParticipantLaunchService publishes the daemon's port. The daemon listens on port 9090 (AGGREGATOR_PORT env var, default) inside its container; since the Admin Monitor process calls it from the host (not from inside another container), the aggregator's docker run now includes -p 9090:9090 — the only Docker participant role that publishes a port. The operator then points the Admin Monitor process at it: AGGREGATOR_API_URL=http:// localhost:9090.

Deleted as dead code: SequentialRoundDispatcher and its test — the single-worker, locally-tracked globalModelVersion counter this daemon used to serialize round processing with (BL-201) no longer applies now that the daemon holds no round state at all; each POST /aggregate call is a self-contained, stateless computation.