Component: Data Component¶
Responsibility¶
Mandatory entry point for all local experiment data. Manages the complete cycle of ingestion via MQTT, persistence in SQLite, and controlled access to the CDM repository.
Core invariant (BL-091):
loadLocalCdmRepository()reads directly fromtimeseries.dband throwsIllegalStateExceptionif there is no observation/event/entity for the queried(clientId, experimentId). MQTT provenance is guaranteed structurally, not by file: the only production writer of the store isMqttTimeSeriesIngestor, which only writes afterenvelope.verify(signatureService)succeeds — so data present in the store already implies it passed signature verification. Thedata/{clientId}/{experimentId}/dataset.csv/metadata.jsonsnapshot (and therequireMqttProvenance()check that used to exist on it) has been removed from this path;dataset()/materializeDatasetFromStore()still exist only as an optional utility for the legacy loaders (Ecg5000DatasetLoader/HarDatasetLoader) and are no longer a prerequisite forloadLocalCdmRepository().
Implementation¶
| Item | Value |
|---|---|
| Module | modules/federated-local-cdm |
| Java package | br.ufma.lsdi.localcdm |
| Main class | DataComponent |
| Default store | SqliteTimeSeriesStore (data/timeseries.db) |
| Production ingestion | MqttTimeSeriesIngestor via the Paho MQTT gateway |
Data lifecycle¶
IoT Adapter (HAR/ECG/DT)
│
│ publish ObservationMessage / EventMessage (MQTT signed)
▼
PahoMqttGateway → MQTT 5 broker (production; the in-memory gateway is a test fixture)
│
│ deliver (wildcard iot/clients/{id}/entities/+/observations)
▼
MqttTimeSeriesIngestor
│ verify signature + payload integrity + experiment/sender scope
│ append(observation)
▼
SqliteTimeSeriesStore → {dataRoot}/timeseries.db
│ (mqtt_observations, mqtt_events, mqtt_entities)
│
│ DataComponent.loadLocalCdmRepository()
│ [throws IllegalStateException if the store is empty for (clientId, experimentId)]
▼
InMemoryLocalCdmRepository
│
│ DataComponent.authorizedTrainingView(clientId, contract, repository, window)
│ [delegates to AuthorizedTrainingViewBuilder.build(...) — same tested logic]
▼
AuthorizedTrainingView → context.authorizedDataView() for TrainingLocalModel
dataset.csv/metadata.json(materialized byDataComponent.materializeDatasetFromStore()) still exist as an optional path, used by the legacy loaders (Ecg5000DatasetLoader/HarDatasetLoader) viadataset()— they are no longer part of theloadLocalCdmRepository()lifecycle, which readsSqliteTimeSeriesStoredirectly.BL-245:
DataComponent.authorizedTrainingView(...)(two static overloads, with/without aTimeWindow) is the literal entry pointFederatedClientServicecalls — before this it callednew AuthorizedTrainingViewBuilder().build(...)directly, bypassingDataComponent. The filtering logic itself is unchanged (delegated, not reimplemented); this only moves the public entry point soDataComponentreally is the single path to authorized data, matching the invariant stated at the top of this document. See 08-local-cdm.md for the filter itself (includingprohibitedFields, now actually enforced) and 08-local-cdm.md for the new closed vocabulary of data types.
MQTT ingestion API¶
subscribeToMqttSource()¶
Registers an MqttTimeSeriesIngestor on the gateway for a specific client/experiment.
DataComponent dc = new DataComponent(ExperimentRuntimePaths.dataRoot());
// Subscribe BEFORE the MQTT replay
dc.subscribeToMqttSource(clientId, experimentId, mqttGateway, signatureService);
Per-client scoping — ClientSenderScope¶
The ingestor subscribes to two groups of topics with different scoping criteria:
| Topic group | Scoping | Why |
|---|---|---|
iot/clients/{clientId}/entities/+/…, mhub/{clientId}/service_topic/+ |
the path itself | clientId is in the topic — the subscription is already exclusive |
microbroker/cddl/event/object/+ |
envelope.senderId() |
CDDL event topics are global: every participant receives every lifecycle event |
Until BL-237, CDDL service/sensor topics (mhub/{clientId}/service_topic/{serviceName}) were
also global (microbroker/cddl/service/+, no clientId in the path), and senderId was the only
thing that said whose message it was — exactly like the event topics still are today. All replay
adapters (HarMqttReplayAdapter, EcgMqttReplayAdapter, CsvDatasetSimulationSource) still build
that senderId via ClientSenderScope (module federated-common), both for the events that still
need it and for audit/signing purposes on service messages:
// publisher
String senderId = ClientSenderScope.senderIdFor("har-sim-", clientId); // "har-sim-1"
// subscriber (still used for CDDL event topics)
ClientSenderScope.belongsTo(senderId, clientId); // segment match, not substring
Fixed defect (predates BL-237, still relevant to event topics): the filter used to be
senderId.contains(clientId). With prefixhar-sim-and subjects 1–30,"har-sim-10".contains("1")is true — client1also ingested subjects 10–19 and 21. SinceSqliteTimeSeriesStorewrites withINSERT OR REPLACEon a key that does not include the sender, and the adapters reuse the sameentityId/timestamp across clients (e.g.window-00000atT0), those foreign rows overwrote the client's own. The resulting local dataset had a plausible size and raised no error, but was a blend of ~12 subjects — which nullifies the non-IID partition the federated experiment is meant to measure. Since BL-237 movedclientIdinto the CDDL service-topic path, this class of bug is now structurally impossible for observations (the broker's own topic routing enforces isolation, exactly like the canonicaliot/clients/{clientId}/…topics always did); it remains possible for CDDL events, which is why the regression test below still reproduces it there. A regression test (MqttTimeSeriesIngestorRoutingTest) publishes 30 subjects (one observation per client's own topic, plus an event-based reproduction of the original bug) and asserts each client keeps exactly its own data.
belongsTo also treats a null senderId as not belonging. Previously a null sender skipped the
filter entirely and the message was ingested by every client.
Rejection filters (order matters)¶
Every envelope passes through four checks, in this order:
envelope.verify(signatureService)— authenticity. First, because only an authentic envelope justifies trusting thesenderId/experimentIdit declares for routing.envelope.verifyPayloadIntegrity()— the payload actually matches the signed hash.verify()alone only proves the hash was signed; harmless while payload and hash share a call stack in memory, exploitable as soon as the payload travels as bytes through a real broker.ClientSenderScope.belongsTo(...)— on CDDL topics only.envelope.experimentId()equals the ingestor's experiment — prevents two concurrent replays from contaminating each other's datasets over the shared CDDL topics.
A rejected envelope is never an exception: on a wildcard subscription, receiving another client's message is the normal case.
IngestionDiagnostics — why nothing arrived¶
Each rejection point increments a counter and records the reason, exposed by
MqttTimeSeriesIngestor.diagnostics():
IngestionDiagnostics d = ingestor.diagnostics();
d.received(); // envelopes delivered to this ingestor
d.ingested(); // persisted in the store
d.droppedInvalidSignature();
d.droppedTamperedPayload();
d.droppedForeignSender();
d.droppedForeignExperiment();
d.droppedIncompatiblePayload();
d.isSilent(); // received == 0 — nothing arrived at all
d.isDroppingEverything(); // received > 0 && ingested == 0
d.healthLevel(); // "ok" | "warn" | "error"
isSilent() and isDroppingEverything() separate two failures with opposite causes: the first points
to publication/subscription (the replay never ran, the broker never connected), the second to routing
or cryptography. Before this, all four rejection points were silent, and a client left with an empty
dataset was indistinguishable from one that never received anything — the failure only surfaced rounds
later as "round without quorum" or unexplained accuracy loss.
Counters rather than samples: a UCI HAR window alone publishes 561 observations, so retaining dropped
messages would cost memory proportional to the dataset. lastDropReason() keeps only the most recent
rejection, which is enough to direct the investigation.
IngestionAudit — cross-process check¶
IngestionDiagnostics lives inside the client process (a Docker container). The web monitor runs
elsewhere and only sees the shared store, so it reconciles two numbers it already has:
List<IngestionAudit.ClientIngestionStatus> audit = IngestionAudit.reconcile(
publishedByClient, // baseline from SimulationRoundEngine
store.summaryByClientAndType(experimentId));
| Condition | healthLevel() |
Meaning |
|---|---|---|
ingested == published |
ok |
full delivery |
0 < ingested < published |
warn |
partial delivery — replay still running, or drops |
published > 0 && ingested == 0 |
error |
the client received nothing |
ingested > published (hasSurplus()) |
error |
cross-client contamination |
Surplus is error, not warn: another client's data in the local dataset invalidates the non-IID
partition and breaches data sovereignty — never a tolerable degradation. Rendered by the monitor's
"🔎 Conferência" tab — see 01-admin-api-monitor-web.md.
materializeDatasetFromStore() (optional, BL-091)¶
Reads from the SQLite store and writes dataset.csv + metadata.json with source=mqtt. No
longer called by the 4 production federated sessions (Mnist/Har/Ecg/MimicV31FederatedSession)
nor by the two real Docker entry points (Ecg5000FederatedExperimentApp/HarFederatedExperimentApp)
— loadLocalCdmRepository() no longer depends on this snapshot. Kept as a utility for the legacy
loaders (Ecg5000DatasetLoader/HarDatasetLoader, which call dataset()/
readCsvTable() directly, outside the generic MQTT pipeline) and for manual debugging.
// After the MQTT replay has published the observations (optional use)
DigitalTwinDataset ds = dc.materializeDatasetFromStore(clientId, experimentId);
Throws IllegalStateException if the store is empty for the client/experiment.
awaitMqttDataAndMaterialize()¶
Synchronous version — blocks until minObservations observations are received or the timeout expires, then materializes.
// With explicit threshold and timeout
MqttIngestionResult result = dc.awaitMqttDataAndMaterialize(
clientId, experimentId, mqttGateway, signatureService,
500L, // minObservations
60L // timeoutSeconds
);
System.out.println("Ingested: " + result.observationsIngested());
// Simplified version (1 observation, 30s timeout)
MqttIngestionResult result = dc.awaitMqttDataAndMaterialize(
clientId, experimentId, mqttGateway, signatureService
);
MqttIngestionResult:
public record MqttIngestionResult(
String clientId,
String experimentId,
long observationsIngested,
long eventsIngested,
boolean thresholdReached,
DigitalTwinDataset dataset
) {}
Time-series API (direct store access)¶
Allows reading data ingested via MQTT without depending on a materialized dataset.csv.
// Basic reads
List<LocalObservation> obs = dc.readObservations(clientId, experimentId);
List<LocalObservation> wind = dc.readObservations(clientId, experimentId,
TimeWindow.of(t0, t1));
List<LocalEvent> evts = dc.readEvents(clientId, experimentId);
List<LocalEvent> eWind = dc.readEvents(clientId, experimentId, TimeWindow.of(t0, t1));
List<LocalEntity> ents = dc.readEntities(clientId, experimentId);
// Counts
int obsCount = dc.observationCount(clientId, experimentId);
int evtCount = dc.eventCount(clientId, experimentId);
boolean has = dc.hasTimeSeriesData(clientId, experimentId);
// TimeSeriesStore summaries and latest samples (SQLite store)
List<DataTypeSummary> summary = dc.timeSeriesSummary(experimentId);
List<LocalObservation> latest = dc.latestObservations(clientId, experimentId,
entityId, propertyCode, 10);
awaitIngestion()¶
DataComponent.awaitIngestion(clientId, experimentId, minObservations, timeoutSeconds) registers subscribeToMqttSource(...) and blocks until minObservations is strictly reached or the timeout expires (returning MqttIngestionResult with thresholdReached() without automatically materializing a CSV snapshot, unlike awaitMqttDataAndMaterialize).
Centralized Differential Privacy (BL-081)¶
Differential privacy is centralized in DataComponent rather than implemented ad hoc inside individual TrainingLocalModel classes:
DataComponent.differentialPrivacyConfig(experimentId)anddifferentialPrivacyConfigFromXml(InputStream)parse<orchestration><privacy>fromresearch/experiments/{experimentId}/research.xmlusing XXE-safeDifferentialPrivacyConfigReader.DataComponent.applyDifferentialPrivacyIfConfigured(weights, experimentId, seed)(and overload takingDifferentialPrivacyConfig) delegates toDifferentialPrivacyMath.clipAndNoise(...)(L2 clipping + deterministic Gaussian noise) and logs a[DP-UTILIDADE]warning whennoiseToSignalRatioindicates noise dominates the signal (noiseDominatesSignal).
ModelArtifactStore¶
br.ufma.lsdi.localcdm.models.ModelArtifactStore persists StoredModel snapshots (local updates and candidate global models) in {dataRoot}/model_artifacts.db (SQLite WAL mode) via saveLocalModel, saveGlobalModel, localModelsFor, globalModelFor, allGlobalModelsFor, and allLocalModelsFor, consumed by the Admin Monitor Results tab (BL-132).
DataTypeSummary — summary by (clientId, dataType):
public record DataTypeSummary(
String clientId, String experimentId,
String dataType, long observationCount,
long eventCount, long entityCount,
Instant firstSeen, Instant lastSeen
) {}
loadLocalCdmRepository() — direct read from timeseries.db (BL-091)¶
// Loads the CDM directly from the store; throws IllegalStateException if there is no ingested data
InMemoryLocalCdmRepository repo = dc.loadLocalCdmRepository(clientId, experimentId);
Behavior:
| Situation | Result |
|---|---|
There are observations, events, or entities in the store for (clientId, experimentId) |
Loaded via TimeSeriesDatasetBuilder, directly from the store |
| Store empty for the queried pair | IllegalStateException — advises using subscribeToMqttSource()/awaitMqttDataAndMaterialize() |
There is no longer a check on metadata.json/dataset.csv. MQTT provenance is guaranteed
structurally: MqttTimeSeriesIngestor is the only production writer of
appendObservation/appendEvent, and it only writes after envelope.verify(signatureService)
returns true — data present in the store already implies a valid MQTT signature.
Time-series store¶
SqliteTimeSeriesStore (only supported backend)¶
Single file: {dataRoot}/timeseries.db — shared by all clients and experiments of the instance. Uses SQLite WAL mode for concurrent reads and writes.
Tables: mqtt_observations, mqtt_events, mqtt_entities.
The former CsvTimeSeriesStore backend and CSV store mode were removed in BL-146 (createDefaultStore accepts only sqlite via HDT_AF_TIMESERIES_STORE_TYPE / TIMESERIES_STORE, logging a warning and falling back to SqliteTimeSeriesStore otherwise). Note that dataset() / materializeDatasetFromStore() validate integer clientIds via ClientExperimentDataStore, whereas SqliteTimeSeriesStore accepts arbitrary string clientIds (such as Android UUIDs).
createDefaultStore() — static factory¶
// Creates the default store (SQLite) for a root directory
TimeSeriesStore store = DataComponent.createDefaultStore(dataRoot);
// Accesses SqliteTimeSeriesStore directly for advanced queries
SqliteTimeSeriesStore sqlite = dc.sqliteStore(); // null if the store is not SQLite
File structure¶
data/
timeseries.db ← SQLite (all clients and experiments) — the only source read by loadLocalCdmRepository()
{clientId}/
{experimentId}/
dataset.csv ← OPTIONAL, only exists if materializeDatasetFromStore()/dataset() are called
metadata.json ← same — no longer influences loadLocalCdmRepository()
Physical isolation in Docker: ClientTimeSeriesExporter (BL-091)¶
Applies to one of the two Docker paths. The project runs containers in two different ways, and only the first uses this exporter:
Path How the client gets data FederatedDockerClientExecutor— ephemeral, one container per rounda filtered timeseries.dbmounted read-only, produced byClientTimeSeriesExporterDockerParticipantLaunchService— persistent participants, created from the Containers tabMQTT only; no host database is ever copied into the participant The monitor's pipeline explainer used to present the exporter as unconditional, which is wrong for persistent participants and misleading precisely when someone is debugging why data has not arrived. Corrected in BL-209.
Since loadLocalCdmRepository() started reading the entire timeseries.db (shared by all
clients), mounting that file directly into a container would violate the physical data isolation
between clients. FederatedDockerClientExecutor solves this by exporting, on every container
launch, a filtered database containing only that client's rows:
// Used internally by FederatedDockerClientExecutor.execute()
Path exportedDb = new ClientTimeSeriesExporter()
.export(sourceDataRoot, destinationDataRoot, clientId, experimentId);
// exportedDb == destinationDataRoot/timeseries.db, mounted at /app/data/timeseries.db:ro
Implementation: ATTACH DATABASE on the source timeseries.db + INSERT INTO table SELECT * FROM
src.table WHERE client_id=? AND experiment_id=? for the 3 tables, using the schema already
defined by SqliteTimeSeriesStore (instantiated at the destination instead of duplicating the
DDL). At the end, it runs PRAGMA wal_checkpoint(TRUNCATE) + PRAGMA journal_mode=DELETE to
leave a single consistent file, without -wal/-shm — necessary because the file will be mounted
read-only, and the default WAL mode could attempt to write to those auxiliary files even on a read.
The destination file is always deleted and rebuilt from scratch on every call — it never
accumulates data from previous exports. After exporting, FederatedDockerClientExecutor verifies
that the filtered database contains at least one observation/event/entity for the client; otherwise
it throws IllegalStateException (equivalent to the old check for the existence of the
data/{clientId}/{experimentId} directory).
Today, only
Ecg5000FederatedExperimentApp/DockerEcgClientAppandHarFederatedExperimentApp/DockerHarClientAppuseFederatedDockerClientExecutor— the other experiments (MNIST, MIMIC) train via*FederatedSessionin-process, without Docker.
Environment variables¶
| Variable | Default | Description |
|---|---|---|
HDT_AF_DATA_ROOT |
data |
Root of the per-client data (DATA_ROOT fallback via ExperimentRuntimePaths) |
HDT_AF_TIMESERIES_STORE_TYPE / TIMESERIES_STORE |
sqlite |
Time-series store selector (sqlite is the only supported implementation) |
SQLite writer persistence and default store caching (BL-334)¶
Under heavy MQTT dataset ingestion (e.g., UCI HAR replay with millions of messages), per-message JDBC open/close overhead and default PRAGMA synchronous=FULL doubled fsync pressure and starved container CPU/I/O:
SqliteTimeSeriesStore(AutoCloseable): maintains a persistent synchronized writer JDBCConnectionand cachedPreparedStatements (insertObservationStmt,upsertEntityStmt,insertEventStmt) per store instance, appliesSQLiteConfig(busyTimeout=5000,SynchronousMode.NORMAL,TempStore.MEMORY) insideconnect()on every connection (since SQLitesynchronousis connection-scoped), recovers cleanly on write errors (recoverFailedWrite()), and registers ajava.lang.ref.Cleaner(WriterResources) so unclosed instances never leak file descriptors.DataComponent.defaultTimeSeriesStore(): caches theSqliteTimeSeriesStoreinstance perDataComponentvia double-checked locking (volatile TimeSeriesStore defaultStore) instead of opening a fresh store instance on everysubscribeToMqttSource, read, or count call.
Mandatory rules¶
TrainingLocalModelnever readstimeseries.db/dataset.csvdirectly — it receivescontext.authorizedDataView().subscribeToMqttSource()must be called before starting the MQTT replay.loadLocalCdmRepository()requires data in the store for(clientId, experimentId)— there is no longer a prerequisite ondataset.csv/metadata.json(BL-091).- No dataset loader writes to
timeseries.dbdirectly — all data passes through the MQTT store (MqttTimeSeriesIngestor). - Docker runs (
FederatedDockerClientExecutor) never mount the sharedtimeseries.db— always a subset generated byClientTimeSeriesExporter.
Tests¶
| File | What it covers |
|---|---|
SqliteTimeSeriesStoreTest.java |
Observations, events, entities, TimeWindow, persistence, summaryByClientAndType |
DataComponentTimeSeriesApiTest.java |
Public API: readObservations, readEvents, readEntities, counts, hasTimeSeriesData, timeSeriesSummary, latestObservations, default SQLite store |
DataComponentLoadLocalCdmRepositoryTest.java |
BL-091: loadLocalCdmRepository() reads directly from the store, throws an exception without data, ignores metadata.json |
ClientTimeSeriesExporterTest.java |
BL-091: filtered export by client/experiment, isolation between clients/experiments, absence of -wal/-shm, re-export without accumulation |
ArchitectureComponentTestSuite.java |
Preflight, MQTT ingestion, loadLocalCdmRepository |