Ir para o conteúdo

Component: Standardized Local Database (Local CDM)

Responsibility

The federated-local-cdm module represents client data in a common model (CDM — Common Data Model), allowing different experiments to use entities, observations, and events consistently. It is the only authorized access to local data — DataComponent is the mandatory entry point.

The module supports only the MQTT path as the mandatory ingestion pipeline: - IoT MQTT replay — observations and events arrive via MQTT and are persisted in SqliteTimeSeriesStore (default). loadLocalCdmRepository() reads this store directly (BL-091) — materializing dataset.csv is no longer a mandatory pipeline step.

Architectural invariant (BL-091): DataComponent.loadLocalCdmRepository() throws IllegalStateException if there is no data in timeseries.db for (clientId, experimentId). MQTT provenance is guaranteed structurally — only MqttTimeSeriesIngestor writes to the store, and only after verifying the envelope's signature — not by a source=mqtt note in metadata.json (that file, when it exists, is just an optional snapshot and is no longer consulted by this check).

Implementation

Item Value
Module modules/federated-local-cdm
Java package br.ufma.lsdi.localcdm
Logical model federated_local_cdm
MVP implementation InMemoryLocalCdmRepository

Main classes (26 types across 27 files)

DataComponent

Mandatory entry point for all local experiment data. Resolves data and differential privacy policies for a given (clientId, experimentId) pair.

DataComponent dataComponent = new DataComponent(dataRoot);

// Production path: subscribe to MQTT ingestion (populates {dataRoot}/timeseries.db)
MqttTimeSeriesIngestor ingestor =
    dataComponent.subscribeToMqttSource(clientId, experimentId, mqttGateway, signatureService);

// Loads the CDM repository directly from timeseries.db (BL-091; throws if empty)
LocalCdmRepository repo = dataComponent.loadLocalCdmRepository(clientId, experimentId);

// Optional legacy CSV snapshot path (not required by loadLocalCdmRepository)
DigitalTwinDataset dataset = dataComponent.materializeDatasetFromStore(clientId, experimentId);

FederatedClientService obtains the contract-filtered training data through DataComponent.authorizedTrainingView(...) (which delegates to AuthorizedTrainingViewBuilder.build(...)).

LocalCdmRepository

Interface that abstracts the client's local repository. The production implementation is InMemoryLocalCdmRepository, built by DataComponent.loadLocalCdmRepository() via TimeSeriesDatasetBuilder directly from timeseries.db (BL-091).

Main operations: - entities() → List<LocalEntity> - observations() → List<LocalObservation> - events() → List<LocalEvent>

InMemoryLocalCdmRepository

In-memory implementation of LocalCdmRepository backed by immutable copies of entities, observations, and events read from timeseries.db (BL-091).

LocalEntity

Represents an entity in the local model (patient, industrial device, activity window). Fields: entityId, entityType, sourceSystem, standardProfile, createdAt, updatedAt, status.

LocalObservation

A time-stamped observation linked to an entity. Fields: observationId, entityId, observedPropertyCode, observedPropertySystem, valueNumeric, valueText, unitUcum, observedAt, provenanceId, qualityScore.

LocalEvent

A discrete event linked to an entity. Fields: eventId, entityId, eventTypeCode, eventTypeSystem, eventStart, severity, provenanceId.

AuthorizedTrainingView

A view of the dataset filtered by the contract's restrictions. Fields: experimentId, clientId, observations (List<LocalObservation>), window (TimeWindow). AuthorizedTrainingViewWriter.writeCsv(...) serializes it to authorized_view.csv (context.authorizedDataView()), which TrainingLocalModel consumes — never the raw repository.

Filters applied by AuthorizedTrainingViewBuilder: - observations whose observedPropertyCode is prohibited by DataSchemaSpec.prohibitedFields are excluded — enforced since BL-245; - observations with qualityScore < minimumQualityScore are discarded; - only allowedObservationCodes are retained (an empty allowedObservationCodes set allows all codes not in prohibitedFields).

prohibitedFields takes precedence over allowedObservationCodes (DataSchemaSpec.permits(code), BL-245) — a code listed in both is excluded.

AuthorizedTrainingViewBuilder

Builder that applies the DataSchemaSpec restrictions (from the contract) to the CDM repository and produces an auditable AuthorizedTrainingView. Since BL-245, invoked through DataComponent.authorizedTrainingView(...) rather than constructed and called directly by FederatedClientService — the logic itself (this class) is unchanged, only the public entry point moved.

DataTypeCatalog — closed data-type vocabulary (BL-245)

br.ufma.lsdi.common.model.DataTypeCatalog (federated-common) — the "predefined table" of data types the architecture supports, requested by the user so a researcher declares contract fields against a shared vocabulary instead of each experiment inventing its own field codes/types freely.

Collection<DataTypeSpec> all = DataTypeCatalog.all();
Optional<DataTypeSpec> spec  = DataTypeCatalog.find("float");
boolean isNative = DataTypeCatalog.isNative("not_a_real_type"); // false

DataTypeSpec(code, displayName, valueKind, defaultUnit, domain, description) — seeded with the 2 generic types that cover 100% of real usage today (float, integer) plus boolean/ categorical/text, ~12 domain-specific codes (heart_rate_bpm, image_pixel_intensity, signal_amplitude_normalized, etc. — consolidation candidates for future experiments, no existing experiment is required to migrate to them), and 4 LGPD exclusion markers (direct_identifier, raw_unprocessed_signal, etc.) for prohibitedFields.

Closed vocabulary, unlike the sibling catalogs it's modeled after (AggregationStrategyCatalog/SelectionStrategyCatalog): DataFieldSpec's compact constructor validates type against DataTypeCatalog.isNative(type) and throws IllegalArgumentException if it isn't found — it is impossible to construct a DataFieldSpec/ExperimentContract with a type outside the catalog in any code path (direct experiment construction, the web form, JSON deserialization). There is no companion Factory/reflection-based custom-type resolution — data types are vocabulary entries, not instantiable strategies, so isNative(code) alone is the entire validation surface. allowedObservationCodes/prohibitedFields (observation codes) are NOT validated against this catalog — they remain free, per-dataset identifiers (MNIST's pixel_0..pixel_783, ECG5000's t001..t140, etc.); only DataFieldSpec.type() (the value type) is closed.

Already usable by federated-client without any dependency change — the module already depends on federated-common — satisfying "the client must be able to query this list" as a plain Java call, no network round-trip required.

DataProfileService & DefaultPreflightValidator

DataProfileService.profile(...) computes LocalDataProfile statistics from the LocalCdmRepository, and DefaultPreflightValidator.validate(contract, roundId, profile) runs the statistical preflight before local training.

Checks performed by DefaultPreflightValidator:

Check Passing condition
minimum_examples_per_client sampleCount ≥ sampleRequirements.minimumExamplesPerClient()
minimum_positive_examples_per_client positiveCount ≥ sampleRequirements.minimumPositiveExamplesPerClient()
minimum_entities_per_client entityCount ≥ sampleRequirements.minimumEntitiesPerClient()
expected_positive_rate_range positive rate within [minExpectedPositiveRate, maxExpectedPositiveRate]
missingness_threshold:{code} missingness rate for required property code ≤ threshold

When a check fails, PreflightResult.failedChecks() records the reasons. The web monitor displays the failed checks as operational diagnostics without exposing raw data.

LocalDataProfile profile = dataProfileService.profile(contract, roundId, cdmRepository);
PreflightResult result = new DefaultPreflightValidator().validate(contract, roundId, profile);
if (!result.eligibleForTraining()) {
    // client does not participate in this round
    log.info("Preflight failed: {}", result.failedChecks());
}

PreflightResult

Record (in br.ufma.lsdi.common.model) with the preflight result:

public record PreflightResult(
    String experimentId,
    String clientId,
    long roundId,
    boolean eligibleForTraining,
    int sampleCount,
    int positiveCount,
    int entityCount,
    List<String> failedChecks
) {}

Additional classes in federated-local-cdm

  • AuthorizedTrainingViewWriter, ClientExperimentDataStore, DigitalTwinDataset
  • DifferentialPrivacyConfig, DifferentialPrivacyConfigReader (XXE-safe XML), DifferentialPrivacyMath (BL-081)
  • br.ufma.lsdi.localcdm.models.ModelArtifactStore & StoredModel (model_artifacts.db)
  • br.ufma.lsdi.localcdm.timeseries: TimeSeriesStore, ClientTimeSeriesExporter, IngestionDiagnostics, IngestionAudit, TimeSeriesDatasetBuilder (note: timeseries/TimeWindow.java is a legacy comment-only stub; TimeWindow moved to br.ufma.lsdi.common.model.TimeWindow)

LocalDataProfile

Statistical profile of the local dataset, written as data_profile.json after the preflight. Contains counts, prevalence, and quality metadata without exposing individual data.


Time series (MQTT ingestion)

MqttTimeSeriesIngestor

Subscribes to the client's IoT/CDDL topics and persists observations and events in SQLite.

iot/clients/{clientId}/entities/{entityId}/observations  → mqtt_observations
iot/clients/{clientId}/entities/{entityId}/events         → mqtt_events

Internal flow: 1. Receives an MqttEnvelope<ObservationMessage> or MqttEnvelope<EventMessage>. 2. Verifies the envelope signature, payload integrity, experiment identity, and sender scope. 3. Persists accepted messages via SqliteTimeSeriesStore.

The ingestor is created by DataComponent.subscribeToMqttSource(...) and registered on the MqttGateway with four topic filters: IotTopics.allObservationsForClient(clientId), IotTopics.allEventsForClient(clientId), IotTopics.cddlServicesForClient(clientId), and IotTopics.cddlAllEvents().

SqliteTimeSeriesStore (only supported backend)

Embedded SQLite database at {dataRoot}/timeseries.db. A database is shared by the clients and experiments using that DataComponent root; records are keyed and queried by client and experiment. Implements AutoCloseable, maintains a persistent writer connection (WriterResources, Cleaner) with SQLiteConfig (busyTimeout=5000, SynchronousMode.NORMAL, TempStore.MEMORY) and WAL mode for concurrent reads and writes (BL-334).

Table Main columns
mqtt_observations id, client_id, experiment_id, entity_id, entity_type, property_code, property_system, value_numeric, value_text, unit_ucum, observed_at, quality_score, ingested_at
mqtt_events id, client_id, experiment_id, entity_id, entity_type, event_type_code, event_type_system, event_start, severity, ingested_at
mqtt_entities client_id, experiment_id, entity_id, entity_type, first_seen, last_seen, observation_count, event_count

Advanced operations: - summaryByClientAndType(experimentId) → List<DataTypeSummary> — statistical summary by (clientId, dataType) - latestObservations(clientId, experimentId, entityId, propertyCode, limit) → sparklines for the monitor - readObservationsByDataType(clientId, experimentId) → observations grouped by dataType

SqliteTimeSeriesStore is the only production backend. The former CSV backend and TIMESERIES_STORE=csv option were removed in BL-146.


Time-series API — DataComponent

DataComponent exposes a public API for reading directly from the store, without needing to access dataset.csv:

DataComponent dc = new DataComponent(dataRoot);

// Basic reads
List<LocalObservation> obs    = dc.readObservations(clientId, experimentId);
List<LocalObservation> windowed = dc.readObservations(clientId, experimentId, TimeWindow.of(t0, t1));
List<LocalEvent>       events  = dc.readEvents(clientId, experimentId);
List<LocalEvent>       eWindow = dc.readEvents(clientId, experimentId, TimeWindow.of(t0, t1));
List<LocalEntity>      ents    = dc.readEntities(clientId, experimentId);

// Counts
int obsCount   = dc.observationCount(clientId, experimentId);
int evtCount   = dc.eventCount(clientId, experimentId);
boolean hasData = dc.hasTimeSeriesData(clientId, experimentId);

// Advanced summaries are provided by the production SQLite store.
List<DataTypeSummary>  summary = dc.timeSeriesSummary(experimentId);
List<LocalObservation> latest  = dc.latestObservations(clientId, experimentId,
                                                        entityId, propertyCode, 10);

DataTypeSummary

Record that aggregates statistics by (clientId, experimentId, dataType):

public record DataTypeSummary(
    String   clientId,
    String   experimentId,
    String   dataType,
    long     observationCount,
    long     eventCount,
    long     entityCount,
    Instant  firstSeen,
    Instant  lastSeen
) {}

Per-client data structure

data/
  timeseries.db            ← SQLite for this DataComponent root; records include client_id and experiment_id
                              THE ONLY source read by loadLocalCdmRepository() (BL-091)
  {client_id}/
    {experiment_id}/
      dataset.csv          ← OPTIONAL — only exists if materializeDatasetFromStore()/dataset()
      metadata.json        ← are called manually; does not influence loadLocalCdmRepository()

metadata.json — when generated manually

{
  "client_id": "3",
  "experiment_id": "exp-har-smartphones-001",
  "dataset_file": "dataset.csv",
  "records": 1200,
  "created_at": "2026-06-26T10:00:00Z",
  "notes": ["source=mqtt", "observations=7352", "events=0"]
}

This file is optional and is no longer consulted by loadLocalCdmRepository() (BL-091) — MQTT provenance is now guaranteed structurally: only MqttTimeSeriesIngestor writes to timeseries.db, and only after verifying the envelope's signature.


Mandatory rules

  • TrainingLocalModel never accesses timeseries.db/dataset.csv directly — it uses context.authorizedDataView().
  • DataComponent is the only authorized entry point to the local dataset.
  • loadLocalCdmRepository() requires data in timeseries.db for (clientId, experimentId) — throws IllegalStateException otherwise; there is no longer a metadata.json check (BL-091).
  • Fields prohibited in DataSchemaSpec.prohibitedFields never enter the AuthorizedTrainingView (an empty allowedObservationCodes set allows all non-prohibited codes).
  • The preflight must run before training; failures block the client for that round.
  • The local profile (data_profile.json) does not expose individualized data — only counts.

Current state

The module uses SqliteTimeSeriesStore by default (embedded database, no server) with support for IoT MQTT replay via MqttTimeSeriesIngestor. The production evolution may migrate to PostgreSQL with the federated_local_cdm schema while keeping the same TimeSeriesStore interface.


Tests

File What it covers
SqliteTimeSeriesStoreTest.java Observations, events, entities, TimeWindow, persistence, summaryByClientAndType
DataComponentTimeSeriesApiTest.java Public API: readObservations, readEvents, readEntities, counts, hasTimeSeriesData, timeSeriesSummary, latestObservations, default SQLite store
ArchitectureComponentTestSuite.java In-memory repository, MQTT ingestion, preflight, materialization