Component: Standardized Local Database (Local CDM)¶
Responsibility¶
The federated-local-cdm module represents client data in a common model (CDM — Common Data Model), allowing different experiments to use entities, observations, and events consistently. It is the only authorized access to local data — DataComponent is the mandatory entry point.
The module supports only the MQTT path as the mandatory ingestion pipeline:
- IoT MQTT replay — observations and events arrive via MQTT and are persisted in SqliteTimeSeriesStore (default). loadLocalCdmRepository() reads this store directly (BL-091) — materializing dataset.csv is no longer a mandatory pipeline step.
Architectural invariant (BL-091):
DataComponent.loadLocalCdmRepository()throwsIllegalStateExceptionif there is no data intimeseries.dbfor(clientId, experimentId). MQTT provenance is guaranteed structurally — onlyMqttTimeSeriesIngestorwrites to the store, and only after verifying the envelope's signature — not by asource=mqttnote inmetadata.json(that file, when it exists, is just an optional snapshot and is no longer consulted by this check).
Implementation¶
| Item | Value |
|---|---|
| Module | modules/federated-local-cdm |
| Java package | br.ufma.lsdi.localcdm |
| Logical model | federated_local_cdm |
| MVP implementation | InMemoryLocalCdmRepository |
Main classes (26 types across 27 files)¶
DataComponent¶
Mandatory entry point for all local experiment data. Resolves data and differential privacy policies for a given (clientId, experimentId) pair.
DataComponent dataComponent = new DataComponent(dataRoot);
// Production path: subscribe to MQTT ingestion (populates {dataRoot}/timeseries.db)
MqttTimeSeriesIngestor ingestor =
dataComponent.subscribeToMqttSource(clientId, experimentId, mqttGateway, signatureService);
// Loads the CDM repository directly from timeseries.db (BL-091; throws if empty)
LocalCdmRepository repo = dataComponent.loadLocalCdmRepository(clientId, experimentId);
// Optional legacy CSV snapshot path (not required by loadLocalCdmRepository)
DigitalTwinDataset dataset = dataComponent.materializeDatasetFromStore(clientId, experimentId);
FederatedClientService obtains the contract-filtered training data through DataComponent.authorizedTrainingView(...) (which delegates to AuthorizedTrainingViewBuilder.build(...)).
LocalCdmRepository¶
Interface that abstracts the client's local repository. The production implementation is InMemoryLocalCdmRepository, built by DataComponent.loadLocalCdmRepository() via TimeSeriesDatasetBuilder directly from timeseries.db (BL-091).
Main operations:
- entities() → List<LocalEntity>
- observations() → List<LocalObservation>
- events() → List<LocalEvent>
InMemoryLocalCdmRepository¶
In-memory implementation of LocalCdmRepository backed by immutable copies of entities, observations, and events read from timeseries.db (BL-091).
LocalEntity¶
Represents an entity in the local model (patient, industrial device, activity window). Fields: entityId, entityType, sourceSystem, standardProfile, createdAt, updatedAt, status.
LocalObservation¶
A time-stamped observation linked to an entity. Fields: observationId, entityId, observedPropertyCode, observedPropertySystem, valueNumeric, valueText, unitUcum, observedAt, provenanceId, qualityScore.
LocalEvent¶
A discrete event linked to an entity. Fields: eventId, entityId, eventTypeCode, eventTypeSystem, eventStart, severity, provenanceId.
AuthorizedTrainingView¶
A view of the dataset filtered by the contract's restrictions. Fields: experimentId, clientId, observations (List<LocalObservation>), window (TimeWindow). AuthorizedTrainingViewWriter.writeCsv(...) serializes it to authorized_view.csv (context.authorizedDataView()), which TrainingLocalModel consumes — never the raw repository.
Filters applied by AuthorizedTrainingViewBuilder:
- observations whose observedPropertyCode is prohibited by DataSchemaSpec.prohibitedFields are excluded — enforced since BL-245;
- observations with qualityScore < minimumQualityScore are discarded;
- only allowedObservationCodes are retained (an empty allowedObservationCodes set allows all codes not in prohibitedFields).
prohibitedFields takes precedence over allowedObservationCodes (DataSchemaSpec.permits(code),
BL-245) — a code listed in both is excluded.
AuthorizedTrainingViewBuilder¶
Builder that applies the DataSchemaSpec restrictions (from the contract) to the CDM repository
and produces an auditable AuthorizedTrainingView. Since BL-245, invoked through
DataComponent.authorizedTrainingView(...) rather than
constructed and called directly by FederatedClientService — the logic itself (this class) is
unchanged, only the public entry point moved.
DataTypeCatalog — closed data-type vocabulary (BL-245)¶
br.ufma.lsdi.common.model.DataTypeCatalog (federated-common) — the "predefined table" of data
types the architecture supports, requested by the user so a researcher declares contract fields
against a shared vocabulary instead of each experiment inventing its own field codes/types freely.
Collection<DataTypeSpec> all = DataTypeCatalog.all();
Optional<DataTypeSpec> spec = DataTypeCatalog.find("float");
boolean isNative = DataTypeCatalog.isNative("not_a_real_type"); // false
DataTypeSpec(code, displayName, valueKind, defaultUnit, domain, description) — seeded with the
2 generic types that cover 100% of real usage today (float, integer) plus boolean/
categorical/text, ~12 domain-specific codes (heart_rate_bpm, image_pixel_intensity,
signal_amplitude_normalized, etc. — consolidation candidates for future experiments, no existing
experiment is required to migrate to them), and 4 LGPD exclusion markers (direct_identifier,
raw_unprocessed_signal, etc.) for prohibitedFields.
Closed vocabulary, unlike the sibling catalogs it's modeled after
(AggregationStrategyCatalog/SelectionStrategyCatalog): DataFieldSpec's compact constructor
validates type against DataTypeCatalog.isNative(type) and throws IllegalArgumentException
if it isn't found — it is impossible to construct a DataFieldSpec/ExperimentContract with a
type outside the catalog in any code path (direct experiment construction, the web form,
JSON deserialization). There is no companion Factory/reflection-based custom-type resolution —
data types are vocabulary entries, not instantiable strategies, so isNative(code) alone is the
entire validation surface. allowedObservationCodes/prohibitedFields (observation codes) are
NOT validated against this catalog — they remain free, per-dataset identifiers (MNIST's
pixel_0..pixel_783, ECG5000's t001..t140, etc.); only DataFieldSpec.type() (the value
type) is closed.
Already usable by federated-client without any dependency change — the module already depends
on federated-common — satisfying "the client must be able to query this list" as a plain Java
call, no network round-trip required.
DataProfileService & DefaultPreflightValidator¶
DataProfileService.profile(...) computes LocalDataProfile statistics from the LocalCdmRepository, and DefaultPreflightValidator.validate(contract, roundId, profile) runs the statistical preflight before local training.
Checks performed by DefaultPreflightValidator:
| Check | Passing condition |
|---|---|
minimum_examples_per_client |
sampleCount ≥ sampleRequirements.minimumExamplesPerClient() |
minimum_positive_examples_per_client |
positiveCount ≥ sampleRequirements.minimumPositiveExamplesPerClient() |
minimum_entities_per_client |
entityCount ≥ sampleRequirements.minimumEntitiesPerClient() |
expected_positive_rate_range |
positive rate within [minExpectedPositiveRate, maxExpectedPositiveRate] |
missingness_threshold:{code} |
missingness rate for required property code ≤ threshold |
When a check fails, PreflightResult.failedChecks() records the reasons. The web monitor displays the failed checks as operational diagnostics without exposing raw data.
LocalDataProfile profile = dataProfileService.profile(contract, roundId, cdmRepository);
PreflightResult result = new DefaultPreflightValidator().validate(contract, roundId, profile);
if (!result.eligibleForTraining()) {
// client does not participate in this round
log.info("Preflight failed: {}", result.failedChecks());
}
PreflightResult¶
Record (in br.ufma.lsdi.common.model) with the preflight result:
public record PreflightResult(
String experimentId,
String clientId,
long roundId,
boolean eligibleForTraining,
int sampleCount,
int positiveCount,
int entityCount,
List<String> failedChecks
) {}
Additional classes in federated-local-cdm¶
AuthorizedTrainingViewWriter,ClientExperimentDataStore,DigitalTwinDatasetDifferentialPrivacyConfig,DifferentialPrivacyConfigReader(XXE-safe XML),DifferentialPrivacyMath(BL-081)br.ufma.lsdi.localcdm.models.ModelArtifactStore&StoredModel(model_artifacts.db)br.ufma.lsdi.localcdm.timeseries:TimeSeriesStore,ClientTimeSeriesExporter,IngestionDiagnostics,IngestionAudit,TimeSeriesDatasetBuilder(note:timeseries/TimeWindow.javais a legacy comment-only stub;TimeWindowmoved tobr.ufma.lsdi.common.model.TimeWindow)
LocalDataProfile¶
Statistical profile of the local dataset, written as data_profile.json after the preflight. Contains counts, prevalence, and quality metadata without exposing individual data.
Time series (MQTT ingestion)¶
MqttTimeSeriesIngestor¶
Subscribes to the client's IoT/CDDL topics and persists observations and events in SQLite.
iot/clients/{clientId}/entities/{entityId}/observations → mqtt_observations
iot/clients/{clientId}/entities/{entityId}/events → mqtt_events
Internal flow:
1. Receives an MqttEnvelope<ObservationMessage> or MqttEnvelope<EventMessage>.
2. Verifies the envelope signature, payload integrity, experiment identity, and sender scope.
3. Persists accepted messages via SqliteTimeSeriesStore.
The ingestor is created by DataComponent.subscribeToMqttSource(...) and registered on the MqttGateway with four topic filters: IotTopics.allObservationsForClient(clientId), IotTopics.allEventsForClient(clientId), IotTopics.cddlServicesForClient(clientId), and IotTopics.cddlAllEvents().
SqliteTimeSeriesStore (only supported backend)¶
Embedded SQLite database at {dataRoot}/timeseries.db. A database is shared by the clients and
experiments using that DataComponent root; records are keyed and queried by client and experiment.
Implements AutoCloseable, maintains a persistent writer connection (WriterResources, Cleaner) with SQLiteConfig (busyTimeout=5000, SynchronousMode.NORMAL, TempStore.MEMORY) and WAL mode for concurrent reads and writes (BL-334).
| Table | Main columns |
|---|---|
mqtt_observations |
id, client_id, experiment_id, entity_id, entity_type, property_code, property_system, value_numeric, value_text, unit_ucum, observed_at, quality_score, ingested_at |
mqtt_events |
id, client_id, experiment_id, entity_id, entity_type, event_type_code, event_type_system, event_start, severity, ingested_at |
mqtt_entities |
client_id, experiment_id, entity_id, entity_type, first_seen, last_seen, observation_count, event_count |
Advanced operations:
- summaryByClientAndType(experimentId) → List<DataTypeSummary> — statistical summary by (clientId, dataType)
- latestObservations(clientId, experimentId, entityId, propertyCode, limit) → sparklines for the monitor
- readObservationsByDataType(clientId, experimentId) → observations grouped by dataType
SqliteTimeSeriesStore is the only production backend. The former CSV backend and
TIMESERIES_STORE=csv option were removed in BL-146.
Time-series API — DataComponent¶
DataComponent exposes a public API for reading directly from the store, without needing to access dataset.csv:
DataComponent dc = new DataComponent(dataRoot);
// Basic reads
List<LocalObservation> obs = dc.readObservations(clientId, experimentId);
List<LocalObservation> windowed = dc.readObservations(clientId, experimentId, TimeWindow.of(t0, t1));
List<LocalEvent> events = dc.readEvents(clientId, experimentId);
List<LocalEvent> eWindow = dc.readEvents(clientId, experimentId, TimeWindow.of(t0, t1));
List<LocalEntity> ents = dc.readEntities(clientId, experimentId);
// Counts
int obsCount = dc.observationCount(clientId, experimentId);
int evtCount = dc.eventCount(clientId, experimentId);
boolean hasData = dc.hasTimeSeriesData(clientId, experimentId);
// Advanced summaries are provided by the production SQLite store.
List<DataTypeSummary> summary = dc.timeSeriesSummary(experimentId);
List<LocalObservation> latest = dc.latestObservations(clientId, experimentId,
entityId, propertyCode, 10);
DataTypeSummary¶
Record that aggregates statistics by (clientId, experimentId, dataType):
public record DataTypeSummary(
String clientId,
String experimentId,
String dataType,
long observationCount,
long eventCount,
long entityCount,
Instant firstSeen,
Instant lastSeen
) {}
Per-client data structure¶
data/
timeseries.db ← SQLite for this DataComponent root; records include client_id and experiment_id
THE ONLY source read by loadLocalCdmRepository() (BL-091)
{client_id}/
{experiment_id}/
dataset.csv ← OPTIONAL — only exists if materializeDatasetFromStore()/dataset()
metadata.json ← are called manually; does not influence loadLocalCdmRepository()
metadata.json — when generated manually¶
{
"client_id": "3",
"experiment_id": "exp-har-smartphones-001",
"dataset_file": "dataset.csv",
"records": 1200,
"created_at": "2026-06-26T10:00:00Z",
"notes": ["source=mqtt", "observations=7352", "events=0"]
}
This file is optional and is no longer consulted by loadLocalCdmRepository() (BL-091) — MQTT
provenance is now guaranteed structurally: only MqttTimeSeriesIngestor writes to
timeseries.db, and only after verifying the envelope's signature.
Mandatory rules¶
TrainingLocalModelnever accessestimeseries.db/dataset.csvdirectly — it usescontext.authorizedDataView().DataComponentis the only authorized entry point to the local dataset.loadLocalCdmRepository()requires data intimeseries.dbfor(clientId, experimentId)— throwsIllegalStateExceptionotherwise; there is no longer ametadata.jsoncheck (BL-091).- Fields prohibited in
DataSchemaSpec.prohibitedFieldsnever enter theAuthorizedTrainingView(an emptyallowedObservationCodesset allows all non-prohibited codes). - The preflight must run before training; failures block the client for that round.
- The local profile (
data_profile.json) does not expose individualized data — only counts.
Current state¶
The module uses SqliteTimeSeriesStore by default (embedded database, no server) with support for IoT MQTT replay via MqttTimeSeriesIngestor. The production evolution may migrate to PostgreSQL with the federated_local_cdm schema while keeping the same TimeSeriesStore interface.
Tests¶
| File | What it covers |
|---|---|
SqliteTimeSeriesStoreTest.java |
Observations, events, entities, TimeWindow, persistence, summaryByClientAndType |
DataComponentTimeSeriesApiTest.java |
Public API: readObservations, readEvents, readEntities, counts, hasTimeSeriesData, timeSeriesSummary, latestObservations, default SQLite store |
ArchitectureComponentTestSuite.java |
In-memory repository, MQTT ingestion, preflight, materialization |