Ir para o conteúdo

Component: Data Component

Responsibility

Mandatory entry point for all local experiment data. Manages the complete cycle of ingestion via MQTT, persistence in SQLite, and controlled access to the CDM repository.

Core invariant (BL-091): loadLocalCdmRepository() reads directly from timeseries.db and throws IllegalStateException if there is no observation/event/entity for the queried (clientId, experimentId). MQTT provenance is guaranteed structurally, not by file: the only production writer of the store is MqttTimeSeriesIngestor, which only writes after envelope.verify(signatureService) succeeds — so data present in the store already implies it passed signature verification. The data/{clientId}/{experimentId}/dataset.csv/metadata.json snapshot (and the requireMqttProvenance() check that used to exist on it) has been removed from this path; dataset()/materializeDatasetFromStore() still exist only as an optional utility for the legacy loaders (Ecg5000DatasetLoader/HarDatasetLoader) and are no longer a prerequisite for loadLocalCdmRepository().

Implementation

Item Value
Module modules/federated-local-cdm
Java package br.ufma.lsdi.localcdm
Main class DataComponent
Default store SqliteTimeSeriesStore (data/timeseries.db)
Production ingestion MqttTimeSeriesIngestor via the Paho MQTT gateway

Data lifecycle

IoT Adapter (HAR/ECG/DT)
    │
    │ publish ObservationMessage / EventMessage (MQTT signed)
    ▼
PahoMqttGateway → MQTT 5 broker (production; the in-memory gateway is a test fixture)
    │
    │ deliver (wildcard iot/clients/{id}/entities/+/observations)
    ▼
MqttTimeSeriesIngestor
    │ verify signature + payload integrity + experiment/sender scope
    │ append(observation)
    ▼
SqliteTimeSeriesStore       → {dataRoot}/timeseries.db
    │                              (mqtt_observations, mqtt_events, mqtt_entities)
    │
    │ DataComponent.loadLocalCdmRepository()
    │ [throws IllegalStateException if the store is empty for (clientId, experimentId)]
    ▼
InMemoryLocalCdmRepository
    │
    │ DataComponent.authorizedTrainingView(clientId, contract, repository, window)
    │ [delegates to AuthorizedTrainingViewBuilder.build(...) — same tested logic]
    ▼
AuthorizedTrainingView      → context.authorizedDataView() for TrainingLocalModel

dataset.csv/metadata.json (materialized by DataComponent.materializeDatasetFromStore()) still exist as an optional path, used by the legacy loaders (Ecg5000DatasetLoader/HarDatasetLoader) via dataset() — they are no longer part of the loadLocalCdmRepository() lifecycle, which reads SqliteTimeSeriesStore directly.

BL-245: DataComponent.authorizedTrainingView(...) (two static overloads, with/without a TimeWindow) is the literal entry point FederatedClientService calls — before this it called new AuthorizedTrainingViewBuilder().build(...) directly, bypassing DataComponent. The filtering logic itself is unchanged (delegated, not reimplemented); this only moves the public entry point so DataComponent really is the single path to authorized data, matching the invariant stated at the top of this document. See 08-local-cdm.md for the filter itself (including prohibitedFields, now actually enforced) and 08-local-cdm.md for the new closed vocabulary of data types.


MQTT ingestion API

subscribeToMqttSource()

Registers an MqttTimeSeriesIngestor on the gateway for a specific client/experiment.

DataComponent dc = new DataComponent(ExperimentRuntimePaths.dataRoot());

// Subscribe BEFORE the MQTT replay
dc.subscribeToMqttSource(clientId, experimentId, mqttGateway, signatureService);

Per-client scoping — ClientSenderScope

The ingestor subscribes to two groups of topics with different scoping criteria:

Topic group Scoping Why
iot/clients/{clientId}/entities/+/…, mhub/{clientId}/service_topic/+ the path itself clientId is in the topic — the subscription is already exclusive
microbroker/cddl/event/object/+ envelope.senderId() CDDL event topics are global: every participant receives every lifecycle event

Until BL-237, CDDL service/sensor topics (mhub/{clientId}/service_topic/{serviceName}) were also global (microbroker/cddl/service/+, no clientId in the path), and senderId was the only thing that said whose message it was — exactly like the event topics still are today. All replay adapters (HarMqttReplayAdapter, EcgMqttReplayAdapter, CsvDatasetSimulationSource) still build that senderId via ClientSenderScope (module federated-common), both for the events that still need it and for audit/signing purposes on service messages:

// publisher
String senderId = ClientSenderScope.senderIdFor("har-sim-", clientId);   // "har-sim-1"

// subscriber (still used for CDDL event topics)
ClientSenderScope.belongsTo(senderId, clientId);   // segment match, not substring

Fixed defect (predates BL-237, still relevant to event topics): the filter used to be senderId.contains(clientId). With prefix har-sim- and subjects 1–30, "har-sim-10".contains("1") is true — client 1 also ingested subjects 10–19 and 21. Since SqliteTimeSeriesStore writes with INSERT OR REPLACE on a key that does not include the sender, and the adapters reuse the same entityId/timestamp across clients (e.g. window-00000 at T0), those foreign rows overwrote the client's own. The resulting local dataset had a plausible size and raised no error, but was a blend of ~12 subjects — which nullifies the non-IID partition the federated experiment is meant to measure. Since BL-237 moved clientId into the CDDL service-topic path, this class of bug is now structurally impossible for observations (the broker's own topic routing enforces isolation, exactly like the canonical iot/clients/{clientId}/… topics always did); it remains possible for CDDL events, which is why the regression test below still reproduces it there. A regression test (MqttTimeSeriesIngestorRoutingTest) publishes 30 subjects (one observation per client's own topic, plus an event-based reproduction of the original bug) and asserts each client keeps exactly its own data.

belongsTo also treats a null senderId as not belonging. Previously a null sender skipped the filter entirely and the message was ingested by every client.

Rejection filters (order matters)

Every envelope passes through four checks, in this order:

  1. envelope.verify(signatureService) — authenticity. First, because only an authentic envelope justifies trusting the senderId/experimentId it declares for routing.
  2. envelope.verifyPayloadIntegrity() — the payload actually matches the signed hash. verify() alone only proves the hash was signed; harmless while payload and hash share a call stack in memory, exploitable as soon as the payload travels as bytes through a real broker.
  3. ClientSenderScope.belongsTo(...) — on CDDL topics only.
  4. envelope.experimentId() equals the ingestor's experiment — prevents two concurrent replays from contaminating each other's datasets over the shared CDDL topics.

A rejected envelope is never an exception: on a wildcard subscription, receiving another client's message is the normal case.

IngestionDiagnostics — why nothing arrived

Each rejection point increments a counter and records the reason, exposed by MqttTimeSeriesIngestor.diagnostics():

IngestionDiagnostics d = ingestor.diagnostics();
d.received();                    // envelopes delivered to this ingestor
d.ingested();                    // persisted in the store
d.droppedInvalidSignature();
d.droppedTamperedPayload();
d.droppedForeignSender();
d.droppedForeignExperiment();
d.droppedIncompatiblePayload();
d.isSilent();                    // received == 0 — nothing arrived at all
d.isDroppingEverything();        // received > 0 && ingested == 0
d.healthLevel();                 // "ok" | "warn" | "error"

isSilent() and isDroppingEverything() separate two failures with opposite causes: the first points to publication/subscription (the replay never ran, the broker never connected), the second to routing or cryptography. Before this, all four rejection points were silent, and a client left with an empty dataset was indistinguishable from one that never received anything — the failure only surfaced rounds later as "round without quorum" or unexplained accuracy loss.

Counters rather than samples: a UCI HAR window alone publishes 561 observations, so retaining dropped messages would cost memory proportional to the dataset. lastDropReason() keeps only the most recent rejection, which is enough to direct the investigation.

IngestionAudit — cross-process check

IngestionDiagnostics lives inside the client process (a Docker container). The web monitor runs elsewhere and only sees the shared store, so it reconciles two numbers it already has:

List<IngestionAudit.ClientIngestionStatus> audit = IngestionAudit.reconcile(
        publishedByClient,                       // baseline from SimulationRoundEngine
        store.summaryByClientAndType(experimentId));
Condition healthLevel() Meaning
ingested == published ok full delivery
0 < ingested < published warn partial delivery — replay still running, or drops
published > 0 && ingested == 0 error the client received nothing
ingested > published (hasSurplus()) error cross-client contamination

Surplus is error, not warn: another client's data in the local dataset invalidates the non-IID partition and breaches data sovereignty — never a tolerable degradation. Rendered by the monitor's "🔎 Conferência" tab — see 01-admin-api-monitor-web.md.

materializeDatasetFromStore() (optional, BL-091)

Reads from the SQLite store and writes dataset.csv + metadata.json with source=mqtt. No longer called by the 4 production federated sessions (Mnist/Har/Ecg/MimicV31FederatedSession) nor by the two real Docker entry points (Ecg5000FederatedExperimentApp/HarFederatedExperimentApp) — loadLocalCdmRepository() no longer depends on this snapshot. Kept as a utility for the legacy loaders (Ecg5000DatasetLoader/HarDatasetLoader, which call dataset()/ readCsvTable() directly, outside the generic MQTT pipeline) and for manual debugging.

// After the MQTT replay has published the observations (optional use)
DigitalTwinDataset ds = dc.materializeDatasetFromStore(clientId, experimentId);

Throws IllegalStateException if the store is empty for the client/experiment.

awaitMqttDataAndMaterialize()

Synchronous version — blocks until minObservations observations are received or the timeout expires, then materializes.

// With explicit threshold and timeout
MqttIngestionResult result = dc.awaitMqttDataAndMaterialize(
    clientId, experimentId, mqttGateway, signatureService,
    500L,   // minObservations
    60L     // timeoutSeconds
);
System.out.println("Ingested: " + result.observationsIngested());

// Simplified version (1 observation, 30s timeout)
MqttIngestionResult result = dc.awaitMqttDataAndMaterialize(
    clientId, experimentId, mqttGateway, signatureService
);

MqttIngestionResult:

public record MqttIngestionResult(
    String clientId,
    String experimentId,
    long   observationsIngested,
    long   eventsIngested,
    boolean thresholdReached,
    DigitalTwinDataset dataset
) {}


Time-series API (direct store access)

Allows reading data ingested via MQTT without depending on a materialized dataset.csv.

// Basic reads
List<LocalObservation> obs   = dc.readObservations(clientId, experimentId);
List<LocalObservation> wind  = dc.readObservations(clientId, experimentId,
                                                    TimeWindow.of(t0, t1));
List<LocalEvent>       evts  = dc.readEvents(clientId, experimentId);
List<LocalEvent>       eWind = dc.readEvents(clientId, experimentId, TimeWindow.of(t0, t1));
List<LocalEntity>      ents  = dc.readEntities(clientId, experimentId);

// Counts
int  obsCount = dc.observationCount(clientId, experimentId);
int  evtCount = dc.eventCount(clientId, experimentId);
boolean has   = dc.hasTimeSeriesData(clientId, experimentId);

// TimeSeriesStore summaries and latest samples (SQLite store)
List<DataTypeSummary> summary = dc.timeSeriesSummary(experimentId);
List<LocalObservation> latest = dc.latestObservations(clientId, experimentId,
                                                       entityId, propertyCode, 10);

awaitIngestion()

DataComponent.awaitIngestion(clientId, experimentId, minObservations, timeoutSeconds) registers subscribeToMqttSource(...) and blocks until minObservations is strictly reached or the timeout expires (returning MqttIngestionResult with thresholdReached() without automatically materializing a CSV snapshot, unlike awaitMqttDataAndMaterialize).

Centralized Differential Privacy (BL-081)

Differential privacy is centralized in DataComponent rather than implemented ad hoc inside individual TrainingLocalModel classes:

  • DataComponent.differentialPrivacyConfig(experimentId) and differentialPrivacyConfigFromXml(InputStream) parse <orchestration><privacy> from research/experiments/{experimentId}/research.xml using XXE-safe DifferentialPrivacyConfigReader.
  • DataComponent.applyDifferentialPrivacyIfConfigured(weights, experimentId, seed) (and overload taking DifferentialPrivacyConfig) delegates to DifferentialPrivacyMath.clipAndNoise(...) (L2 clipping + deterministic Gaussian noise) and logs a [DP-UTILIDADE] warning when noiseToSignalRatio indicates noise dominates the signal (noiseDominatesSignal).

ModelArtifactStore

br.ufma.lsdi.localcdm.models.ModelArtifactStore persists StoredModel snapshots (local updates and candidate global models) in {dataRoot}/model_artifacts.db (SQLite WAL mode) via saveLocalModel, saveGlobalModel, localModelsFor, globalModelFor, allGlobalModelsFor, and allLocalModelsFor, consumed by the Admin Monitor Results tab (BL-132).

DataTypeSummary — summary by (clientId, dataType):

public record DataTypeSummary(
    String  clientId,       String  experimentId,
    String  dataType,       long    observationCount,
    long    eventCount,     long    entityCount,
    Instant firstSeen,      Instant lastSeen
) {}


loadLocalCdmRepository() — direct read from timeseries.db (BL-091)

// Loads the CDM directly from the store; throws IllegalStateException if there is no ingested data
InMemoryLocalCdmRepository repo = dc.loadLocalCdmRepository(clientId, experimentId);

Behavior:

Situation Result
There are observations, events, or entities in the store for (clientId, experimentId) Loaded via TimeSeriesDatasetBuilder, directly from the store
Store empty for the queried pair IllegalStateException — advises using subscribeToMqttSource()/awaitMqttDataAndMaterialize()

There is no longer a check on metadata.json/dataset.csv. MQTT provenance is guaranteed structurally: MqttTimeSeriesIngestor is the only production writer of appendObservation/appendEvent, and it only writes after envelope.verify(signatureService) returns true — data present in the store already implies a valid MQTT signature.


Time-series store

SqliteTimeSeriesStore (only supported backend)

Single file: {dataRoot}/timeseries.db — shared by all clients and experiments of the instance. Uses SQLite WAL mode for concurrent reads and writes.

Tables: mqtt_observations, mqtt_events, mqtt_entities.

The former CsvTimeSeriesStore backend and CSV store mode were removed in BL-146 (createDefaultStore accepts only sqlite via HDT_AF_TIMESERIES_STORE_TYPE / TIMESERIES_STORE, logging a warning and falling back to SqliteTimeSeriesStore otherwise). Note that dataset() / materializeDatasetFromStore() validate integer clientIds via ClientExperimentDataStore, whereas SqliteTimeSeriesStore accepts arbitrary string clientIds (such as Android UUIDs).


createDefaultStore() — static factory

// Creates the default store (SQLite) for a root directory
TimeSeriesStore store = DataComponent.createDefaultStore(dataRoot);

// Accesses SqliteTimeSeriesStore directly for advanced queries
SqliteTimeSeriesStore sqlite = dc.sqliteStore(); // null if the store is not SQLite

File structure

data/
  timeseries.db               ← SQLite (all clients and experiments) — the only source read by loadLocalCdmRepository()
  {clientId}/
    {experimentId}/
      dataset.csv             ← OPTIONAL, only exists if materializeDatasetFromStore()/dataset() are called
      metadata.json           ← same — no longer influences loadLocalCdmRepository()

Physical isolation in Docker: ClientTimeSeriesExporter (BL-091)

Applies to one of the two Docker paths. The project runs containers in two different ways, and only the first uses this exporter:

Path How the client gets data
FederatedDockerClientExecutor — ephemeral, one container per round a filtered timeseries.db mounted read-only, produced by ClientTimeSeriesExporter
DockerParticipantLaunchService — persistent participants, created from the Containers tab MQTT only; no host database is ever copied into the participant

The monitor's pipeline explainer used to present the exporter as unconditional, which is wrong for persistent participants and misleading precisely when someone is debugging why data has not arrived. Corrected in BL-209.

Since loadLocalCdmRepository() started reading the entire timeseries.db (shared by all clients), mounting that file directly into a container would violate the physical data isolation between clients. FederatedDockerClientExecutor solves this by exporting, on every container launch, a filtered database containing only that client's rows:

// Used internally by FederatedDockerClientExecutor.execute()
Path exportedDb = new ClientTimeSeriesExporter()
    .export(sourceDataRoot, destinationDataRoot, clientId, experimentId);
// exportedDb == destinationDataRoot/timeseries.db, mounted at /app/data/timeseries.db:ro

Implementation: ATTACH DATABASE on the source timeseries.db + INSERT INTO table SELECT * FROM src.table WHERE client_id=? AND experiment_id=? for the 3 tables, using the schema already defined by SqliteTimeSeriesStore (instantiated at the destination instead of duplicating the DDL). At the end, it runs PRAGMA wal_checkpoint(TRUNCATE) + PRAGMA journal_mode=DELETE to leave a single consistent file, without -wal/-shm — necessary because the file will be mounted read-only, and the default WAL mode could attempt to write to those auxiliary files even on a read.

The destination file is always deleted and rebuilt from scratch on every call — it never accumulates data from previous exports. After exporting, FederatedDockerClientExecutor verifies that the filtered database contains at least one observation/event/entity for the client; otherwise it throws IllegalStateException (equivalent to the old check for the existence of the data/{clientId}/{experimentId} directory).

Today, only Ecg5000FederatedExperimentApp/DockerEcgClientApp and HarFederatedExperimentApp/DockerHarClientApp use FederatedDockerClientExecutor — the other experiments (MNIST, MIMIC) train via *FederatedSession in-process, without Docker.


Environment variables

Variable Default Description
HDT_AF_DATA_ROOT data Root of the per-client data (DATA_ROOT fallback via ExperimentRuntimePaths)
HDT_AF_TIMESERIES_STORE_TYPE / TIMESERIES_STORE sqlite Time-series store selector (sqlite is the only supported implementation)

SQLite writer persistence and default store caching (BL-334)

Under heavy MQTT dataset ingestion (e.g., UCI HAR replay with millions of messages), per-message JDBC open/close overhead and default PRAGMA synchronous=FULL doubled fsync pressure and starved container CPU/I/O:

  • SqliteTimeSeriesStore (AutoCloseable): maintains a persistent synchronized writer JDBC Connection and cached PreparedStatements (insertObservationStmt, upsertEntityStmt, insertEventStmt) per store instance, applies SQLiteConfig (busyTimeout=5000, SynchronousMode.NORMAL, TempStore.MEMORY) inside connect() on every connection (since SQLite synchronous is connection-scoped), recovers cleanly on write errors (recoverFailedWrite()), and registers a java.lang.ref.Cleaner (WriterResources) so unclosed instances never leak file descriptors.
  • DataComponent.defaultTimeSeriesStore(): caches the SqliteTimeSeriesStore instance per DataComponent via double-checked locking (volatile TimeSeriesStore defaultStore) instead of opening a fresh store instance on every subscribeToMqttSource, read, or count call.

Mandatory rules

  • TrainingLocalModel never reads timeseries.db/dataset.csv directly — it receives context.authorizedDataView().
  • subscribeToMqttSource() must be called before starting the MQTT replay.
  • loadLocalCdmRepository() requires data in the store for (clientId, experimentId) — there is no longer a prerequisite on dataset.csv/metadata.json (BL-091).
  • No dataset loader writes to timeseries.db directly — all data passes through the MQTT store (MqttTimeSeriesIngestor).
  • Docker runs (FederatedDockerClientExecutor) never mount the shared timeseries.db — always a subset generated by ClientTimeSeriesExporter.

Tests

File What it covers
SqliteTimeSeriesStoreTest.java Observations, events, entities, TimeWindow, persistence, summaryByClientAndType
DataComponentTimeSeriesApiTest.java Public API: readObservations, readEvents, readEntities, counts, hasTimeSeriesData, timeSeriesSummary, latestObservations, default SQLite store
DataComponentLoadLocalCdmRepositoryTest.java BL-091: loadLocalCdmRepository() reads directly from the store, throws an exception without data, ignores metadata.json
ClientTimeSeriesExporterTest.java BL-091: filtered export by client/experiment, isolation between clients/experiments, absence of -wal/-shm, re-export without accumulation
ArchitectureComponentTestSuite.java Preflight, MQTT ingestion, loadLocalCdmRepository