Ir para o conteúdo

Component: Generic Dataset Simulation Engine (BL-087/BL-088/BL-089)

Responsibility

Before this delivery, each dataset (MNIST, MIMIC, ECG5000, HAR) needed its own {Dataset}MqttSimulationAdapter that decoded its raw CSV, partitioned the rows across clients, and built the in-memory CDM repository — Java code duplicated per dataset, even when the input format was a common tabular CSV with a header. In addition, SimulationDataPackageDescriptor (BL-074) already declared numClients/numRounds/ partitionStrategy in the .hdtsim.zip package, but no code read those fields back to configure an executable simulation — they were just documentation read by the researcher.

This component solves both problems with a dataset-agnostic simulation engine: given a tabular CSV with a header and a declarative description of its columns (CsvTabularDatasetSchema), the engine decodes, partitions across clients, and publishes over MQTT with no dataset-specific Java code at all. DataComponent and the federated client never know that the generic engine exists — they continue to receive data exclusively via DataComponent.subscribeToMqttSource/materializeDatasetFromStore/loadLocalCdmRepository (BL-084/BL-085), with no contract change on that end.


Implementation

Item Value
Schema/descriptor module modules/federated-experiment-packager
Simulation engine module modules/federated-simulator
Schema package br.ufma.lsdi.packager.simulation
Engine package br.ufma.lsdi.simulator
Main classes CsvTabularDatasetSchema, DatasetSimulationConfig, DatasetSimulationSource (interface), CsvDatasetSimulationSource (reference implementation), SimulationDataPackagePlayer
Web monitor integration SimulationRoundEngine (datasetType="csv") — see 01-admin-api-monitor-web.md
Migrated pilot research/experiments/exp-mnist-digits-001 (MnistMqttSimulationAdapter)
CLI/script SimulationDataPackagerCli (optional 14-argument mode), scripts/package-simulation-data.sh (WITH_CSV_SCHEMA=1)
Deployment/execution HTTP API (BL-089) DatasetDeploymentService + routes in AdminWebServer — see §10

ADR-004 preserved: federated-simulator now depends on federated-experiment-packager (standalone, depends only on federated-common) to reuse DatasetPartitioner and the new schema — this does not create a cycle with federated-rest-api (the ADR-004 rule specifically forbids simulator → admin-api, and is never forbidden in the opposite direction or for other modules).


1. CsvTabularDatasetSchema — what the CSV columns are

Immutable record (federated-experiment-packager) that describes the format of a tabular CSV dataset, with no knowledge of its content:

public record CsvTabularDatasetSchema(
    String targetColumn,       // target column (label/target) — required
    String featureColumnsSpec, // feature columns — required
    String clientIdColumn,     // column identifying the client — optional
    String recordIdColumn      // column identifying the record — optional
)
  • targetColumn and featureColumnsSpec are required — blank throws FederatedException(ErrorCode.EXPERIMENT_PACKAGE_INVALID, ...).
  • clientIdColumn/recordIdColumn that are null or blank become an empty string (""); their presence is tested via hasClientIdColumn()/hasRecordIdColumn().

featureColumnsSpec notation

A comma-separated list and/or the compact range notation prefixSTART..prefixEND (the same notation already used informally in HAR/ECG/MIMIC/MNIST manifest.yaml files, now formalized and parseable via CsvTabularDatasetSchema.featureColumns()):

new CsvTabularDatasetSchema("label", "pixel_0..pixel_783", "client_id", "record_id")
    .featureColumns();
// → ["pixel_0", "pixel_1", ..., "pixel_783"]  (784 columns, no zero-padding)

new CsvTabularDatasetSchema("classe", "t001..t140", "", "")
    .featureColumns();
// → ["t001", "t002", ..., "t140"]  (zero-padding of width 3, preserved from the starting token)

new CsvTabularDatasetSchema("label", "subject_id,pixel_0..pixel_2,extra", "", "")
    .featureColumns();
// → ["subject_id", "pixel_0", "pixel_1", "pixel_2", "extra"]  (list and range combined)

The numeric width (zero-padding) is always inferred from the token to the left of ..; the two sides of the range must share the same textual prefix, otherwise featureColumns() throws IllegalArgumentException.

Persistence (dataset-schema.json)

Optionally written inside the .hdtsim.zip package, alongside simulation-descriptor.json (SimulationDataPackageDescriptor, BL-074). Hand-rolled JSON (the same style used throughout the rest of the project — no external library). SimulationDataPackageWriter/Reader iterate generically over all files in the package — no change was needed in them to support this new optional file.


2. Client identification — existing column or synthetic partitioning

This is the central point for anyone who needs to separate each client's data: the engine supports two complementary strategies, chosen via DatasetSimulationConfig.partitionStrategy().

(a) Identify the client field in the dataset itself — by_existing_column

Use this when the CSV already has a column identifying the client (e.g., subject_id, client_id, patient_id). Configure CsvTabularDatasetSchema.clientIdColumn with that column's name:

new CsvTabularDatasetSchema("label", "pixel_0..pixel_783", "client_id", "record_id");

When reading the CSV, each row carries the raw value of that column (CsvRow.clientId()). During publication, CsvDatasetSimulationSource groups the rows by that column's value:

for each row i in the dataset:
    clientId = row.clientId()
    if clientId is null/empty OR not among the configured clientIds:
        clientId = clientIds[i % totalClients]     // round-robin fallback (see below)
    assign the row to that clientId

Built-in automatic fallback: if a specific row has no client value (empty cell) or the value does not match any clientId configured in DatasetSimulationConfig.clientIds(), that individual row is assigned a synthetic client via round-robin — not every row needs to have the column filled in; rows with and without client identification can coexist in the same CSV. This is exactly the same algorithm already used by MnistMqttSimulationAdapter since BL-084, now generalized.

(b) Fully synthetic partitioning — sequential_chunk / random_shuffle_seed

Use this when the CSV has no client column at all — the data from a single file (e.g., a sensor CSV with no device identification) must be artificially divided among N simulated clients. In this case, any value in clientIdColumn is entirely ignored — the partitioning ignores row content and uses only the total count, delegating to DatasetPartitioner (federated-experiment-packager, existing since BL-074, but never consumed by any code before BL-088):

DatasetPartitioner.partitionClients(
    "sequential_chunk",      // or "random_shuffle_seed"
    totalRows, clientIds, partitionSeed);
// → Map<clientId, List<Integer>> (row indices, no overlap, covering all rows)
  • sequential_chunk: contiguous blocks in the file's original order (client 1 = rows [0, chunkSize), client 2 = [chunkSize, 2*chunkSize), etc.; the last client absorbs the remainder when the division is not exact).
  • random_shuffle_seed: shuffles the indices deterministically (Random(partitionSeed)) before slicing into contiguous blocks — same deterministic coverage, different order.

Choosing between the two

Situation partitionStrategy clientIdColumn
CSV already identifies the client per row (e.g., subject_id) by_existing_column column name
CSV has no client column — split artificially sequential_chunk or random_shuffle_seed "" (ignored)
CSV has the column, but some rows have no value by_existing_column column name (round-robin fallback covers rows without a value)

stratified_by_label (one of the 4 strategies documented in SimulationDataPackageDescriptor, see 14-iot-replay-simulator.md and the BL-074 descriptor) is not supported by the generic engine in this delivery — it requires external grouping by class label; DatasetSimulationConfig explicitly rejects this strategy with a FederatedException. Datasets that require stratified partitioning still require a dataset-specific Java adapter.


3. DatasetSimulationConfig / DatasetSimulationSource — the data-component interface

DatasetSimulationConfig is the immutable, validated record that replaces hardcoded per-adapter constants (e.g., MNIST's old DEFAULT_CLIENT_IDS):

public record DatasetSimulationConfig(
    String experimentId,
    List<String> clientIds,        // no duplicates, no blank entries
    int totalRounds,                // > 0
    String partitionStrategy,       // by_existing_column | sequential_chunk | random_shuffle_seed
    long partitionSeed,             // ignored outside random_shuffle_seed
    String senderIdPrefix,          // never null (empty by default)
    Map<String, String> extraParameters  // free-form — see train/test/validation segmentation (§4)
)

DatasetSimulationSource is the interface that any simulation engine must implement:

public interface DatasetSimulationSource {
    List<MqttDatasetReplaySimulator.ReplaySummary> simulateAll(
        DatasetSimulationConfig config, MqttGateway mqttGateway, HmacSignatureService signatureService);

    List<MqttDatasetReplaySimulator.ReplaySummary> simulateRound(
        DatasetSimulationConfig config, int roundNumber, MqttGateway mqttGateway,
        HmacSignatureService signatureService);
}

It receives a concrete HmacSignatureService (not the SignatureService interface) because the simulation always needs to register a deterministic identity per client (registerIdentity) before signing — the same pattern already used by the dataset-specific adapters.


4. CsvDatasetSimulationSource — reference implementation

Decodes rows (from a real CSV file or from an in-memory CsvRow list — an extension point for a dataset-specific synthetic fallback, see §6), partitions per §2, builds the in-memory CDM repository, and delegates all MQTT publication to MqttDatasetReplaySimulator.replayRepository(...) — no new transport, the same signed envelope (MqttEnvelope/MqttGateway) used throughout the rest of the project.

CsvTabularDatasetSchema schema = new CsvTabularDatasetSchema(
    "label", "pixel_0..pixel_783", "client_id", "record_id");

CsvDatasetSimulationSource source = CsvDatasetSimulationSource.fromCsvFile(schema, csvFile);

DatasetSimulationConfig config = new DatasetSimulationConfig(
    "exp-meu-experimento-001", List.of("1", "2", "3"), 5,
    "by_existing_column", 0L, "meu-sim-", Map.of());

List<MqttDatasetReplaySimulator.ReplaySummary> resumo =
    source.simulateAll(config, mqttGateway, signatureService);

The published CDM vocabulary (entity/event) is deliberately generic (csv_tabular_record/csv_record_labeled/csv) — the observed-property codes (what actually matters for reconstructing features during training) always come from schema.featureColumns()/schema.targetColumn(), preserving names like pixel_0..pixel_783 intact between the source CSV and what reaches the client over MQTT.

simulateAll vs simulateRound

  • simulateAll publishes, for each client, all the rows that belong to it — covers the entire dataset in a single call.
  • simulateRound(config, roundNumber, ...) publishes only the slice for round roundNumber (1-indexed) out of config.totalRounds(), via DatasetPartitioner.roundSlice (the same slicing already used by HarMqttReplayAdapter/EcgMqttReplayAdapter: chunkSize = max(1, total/totalRounds), the last round absorbs the remainder).

5. Train/test/validation segmentation (BL-088)

Optional support for splitting the dataset into three subsets by a configurable ratio, without changing the already-tested signature of DatasetSimulationConfig — the ratios are passed via extraParameters, well-known keys read only by CsvDatasetSimulationSource:

Key Required? Description
trainRatio Yes, if any of the 3 is present Fraction of rows for training, [0.0, 1.0]
testRatio Yes, if any of the 3 is present Fraction of rows for testing, [0.0, 1.0]
validationRatio Yes, if any of the 3 is present Fraction of rows for validation, [0.0, 1.0]
splitSeed No Shuffle seed; uses partitionSeed if absent
Map<String, String> extra = Map.of(
    "trainRatio", "0.7", "testRatio", "0.15", "validationRatio", "0.15");
DatasetSimulationConfig config = new DatasetSimulationConfig(
    "exp-001", List.of("1", "2"), 5, "by_existing_column", 0L, "sim-", extra);

Rules:

  • The 3 keys must appear together. If any is missing, no segmentation is applied — behavior before BL-088 is preserved by default (all rows go to the clients).
  • The 3 ratios must sum to 1.0 (tolerance 1e-6); otherwise, CsvDatasetSimulationSource throws IllegalArgumentException.
  • Explicit user decision: only validation is retained as a central holdout — it is never published over MQTT. train and test are combined and distributed normally to the clients (both "go to the clients").
  • The validation subset is accessible via source.validationRows(config) — deterministic (the same config/seed always produces the same result), useful for central evaluation by the researcher outside the MQTT flow.
List<CsvDatasetSimulationSource.CsvRow> holdout = source.validationRows(config);
// validation rows — never published, available for central evaluation

6. SimulationDataPackagePlayer — from a .hdtsim.zip package to an executable simulation

Closes the BL-074 gap: before this class, the numClients/numRounds/ partitionStrategy fields of SimulationDataPackageDescriptor were only documentation. The player reads the descriptor plus the dataset-schema.json from an already-extracted directory (via SimulationDataPackageReader) and assembles a ready-to-run simulation:

List<MqttDatasetReplaySimulator.ReplaySummary> resumo = SimulationDataPackagePlayer.simulateAll(
    extractedDirectory, descriptor, mqttGateway, signatureService, "player-sim-");
// default numeric clientIds "1".."numClients" when not explicitly provided

load(...) throws FederatedException(ErrorCode.EXPERIMENT_PACKAGE_INVALID, ...) if dataset-schema.json does not exist in the extracted directory — required so the generic engine knows how to decode the package's CSV. Datasets that require dataset-specific decoding (non-tabular formats, or their own synthetic fallback) continue to use their own {Dataset}MqttSimulationAdapter instead of this player.


7. Packaging CLI and script

SimulationDataPackagerCli optionally accepts 4 more arguments (14 total) to also write dataset-schema.json to the staging directory:

java -cp ... br.ufma.lsdi.packager.simulation.SimulationDataPackagerCli \
  <staging> <output-zip> <experimentId> <datasetEntry> <datasetDescription> <partitionStrategy> \
  <partitionDescription> <numClients> <numRounds> <samplesPerClient> \
  <targetColumn> <featureColumnsSpec> <clientIdColumn> <recordIdColumn>   # optional

scripts/package-simulation-data.sh exposes this interactively via WITH_CSV_SCHEMA=1:

WITH_CSV_SCHEMA=1 sh scripts/package-simulation-data.sh exp-meu-experimento-001 \
  dataset.csv 3 5 by_existing_column
# Interactively prompts for: targetColumn, featureColumnsSpec, clientIdColumn, recordIdColumn

Without WITH_CSV_SCHEMA=1, the package remains valid (legacy mode, descriptor only from BL-074) but requires a dataset-specific Java adapter to be simulated.


8. Web monitor integration

The monitor's round-based simulation session ("▶ Session", SimulationRoundEngine/ AdminWebServer — see 01-admin-api-monitor-web.md and 14-iot-replay-simulator.md) gained datasetType="csv", alongside har/ecg (which continue to use their untouched dataset-specific adapters). Since SimulationSession is a record with many production/test call sites, the CSV schema and the train/test/validation segmentation were not added to it — they are registered separately, before starting the session:

roundEngine.registerCsvSchema(experimentId, schema);              // required for datasetType=csv
roundEngine.registerCsvSplitRatios(experimentId, splitParams);     // optional
roundEngine.registerClientIds(experimentId, expectedClientIds);    // BL-207 — see below
int holdout = roundEngine.csvValidationRowCount(session);          // -1 if not applicable

BL-207 — registerClientIds is not optional in practice. Without it the CSV replay generates synthetic ids "1".."maxClients". Since the published senderId is csv-sim-{clientId} and MqttTimeSeriesIngestor only accepts its own client's sender, publishing to synthetic ids when the registered clients carry different ids (say subject-07) means publishing to clients nobody is listening as — the real clients end up with an empty dataset and no error anywhere. AdminWebServer now calls it at session start with AdminMonitoringService.expectedClientIds(...). POST /experiments/{id}/run-with-dataset already resolved client ids correctly; only the simulation session did not. HAR/ECG do not use this registry — their client ids come from the dataset's own subjects, which is the correct source for them.

The session-creation form (/experiments/{id}/sim-session) exposes the "CSV — Generic Engine" option with conditional fields (targetColumn/featureColumnsSpec/ clientIdColumn/recordIdColumn plus the 3 split ratios); a csv-schema-required banner covers missing required fields; the active session page shows the retained validation holdout count, when applicable.


9. Migrated pilot — MnistMqttSimulationAdapter

MnistMqttSimulationAdapter.simulate(datasetRoot, clientIds, mqttGateway, signatureService, senderIdPrefix) kept its public signature unchanged, but now internally delegates to the generic engine. The only MNIST-specific code left in the adapter is:

  1. Locating the real CSV (mnist_train.csv/dataset.csv);
  2. The deterministic synthetic-fallback formula (10 images per client, one per digit 0-9) — always dataset-specific, never generic.
List<CsvDatasetSimulationSource.CsvRow> rows =
    CsvDatasetSimulationSource.readRowsFromCsv(SCHEMA, resolveDatasetFile(resolvedRoot));
if (rows.isEmpty()) {
    rows = syntheticRows(clientIds);   // the only MNIST-specific piece
}
CsvDatasetSimulationSource source = new CsvDatasetSimulationSource(SCHEMA, rows);

MIMIC/ECG/HAR were not migrated in this delivery (the same incremental-scope decision as BL-084/085) — direct candidates, since they already have CSV tabular loaders structurally equivalent to MNIST's.


10. Dataset deployment/execution HTTP API (BL-089)

Closes the final gap in the cycle: until now, publishing data from a .hdtsim.zip package over MQTT required Java code calling SimulationDataPackagePlayer directly, or the manual flow of the round-based simulation session (§8). This API exposes upload and execution over HTTP, for any experiment with a dataset package declaring dataset-schema.json — it is not MNIST-specific.

DatasetDeploymentService — deployment (POST /datasets/deploy)

Mirrors ExperimentDeploymentService/POST /experiments/deploy (ADR-005), but for data, not code: it extracts the zip with the same protections as SimulationDataPackageReader (zip slip, zip bomb, SHA256SUMS checksum) and loads and executes no code at all. Unlike experiment deployment, it does not deduplicate by content hash — each upload receives a datasetPackageId (UUID) and an isolated directory under ExperimentRuntimePaths.deployedDatasetDirectory(experimentId, datasetPackageId) (root configurable via HDT_AF_DEPLOYED_DATASETS_ROOT/ DEPLOYED_DATASETS_ROOT, default var/deployed-datasets/ — legacy fallback target/deployed-datasets/), allowing resubmissions (new versions of the data), all traceable and available for execution.

curl -F "package=@exp-meu-experimento-001.hdtsim.zip" https://localhost:8088/datasets/deploy

Immediately rejects (FederatedException(EXPERIMENT_PACKAGE_INVALID)) packages without dataset-schema.json — a fast, actionable failure at upload time, instead of only at execution time.

Routes:

Route Method Description
/datasets/deploy GET Upload form
/datasets/deploy POST (multipart) Deploys the package; redirects to /experiments/{id}/datasets
/experiments/{id}/datasets GET Lists the packages deployed for the experiment, with an execution form per package
/experiments/{id}/run-with-dataset POST Executes rounds from a deployed package (see below)
/api/experiments/{id}/datasets GET JSON of the deployed packages

POST /experiments/{id}/run-with-dataset — dedicated execution

A new route, independent of POST /experiments/{id}/run-rounds — it does not change that method or the round-based simulation session flow (§8); it merely orchestrates them in sequence:

for round in 1..numRounds:
    SimulationDataPackagePlayer.load(...).simulateRound(config, round, mqttGateway, signatureService)
        → publishes the round's slice over MQTT (the same signed envelope as always)
    AdminMonitoringService.runRounds(experimentId, contractVersion, 1, fineTune only on the last round)
        → triggers exactly one federated round (session.executeRound)

Form parameters (urlencoded, not multipart — the package was already uploaded via /datasets/deploy):

Field Required Description
datasetPackageId Yes Id returned by /datasets/deploy
contractVersion No (default 1.0.0) Contract version to execute
numRounds No (default = the descriptor's numRounds) Rounds to execute, capped at the total declared in the package
fineTune No If checked, runs +5 fine-tuning rounds after the last requested round

clientIds is resolved automatically from AdminMonitoringService.expectedClientIds(experimentId, contractVersion) (clients already registered on the experiment); if none are registered, it falls back to "1".."numClients" from the package's descriptor.

Session prerequisite: before publishing the first round, the route calls runRounds(experimentId, contractVersion, 0, false) — this creates the experiment session (and, with it, the clients' MQTT subscription) without executing any round, ensuring the clients are already subscribed when the first data point is published. Requires a MqttGateway configured in the monitor (the same requirement as datasetType="csv" in §8) — without it, the route throws IllegalStateException.

Not verified end-to-end in this delivery: the full integration (upload → MQTT publish → round execution → aggregation) depends on Maven infrastructure unavailable in this development environment (see the verification note in backlog.md, BL-089). The logic was reviewed statically, line by line, against the real signatures of SimulationDataPackagePlayer, AdminMonitoringService.runRounds, and DatasetSimulationSource, but running sh scripts/run-tests.sh --module federated-rest-api and a manual end-to-end test is recommended before production use.


Tests

Module Coverage
federated-experiment-packager CsvTabularDatasetSchemaTest (9: validation, range expansion with/without padding, list+range mix, JSON round-trip) · SimulationDataPackagerCliTest (2: legacy mode vs. mode with schema)
federated-simulator DatasetSimulationConfigTest (11: validation of each field, defaults, immutability) · CsvDatasetSimulationSourceTest (per-record counts, per-client identity, round-robin fallback, sequential_chunk, simulateRound/slicing, invalid roundNumber, TrainTestValidationSplitTests with 5 cases) · SimulationDataPackagePlayerTest (3: end-to-end simulation from an extracted package, error without dataset-schema.json, buildConfig defaults)
federated-rest-api SimulationRoundEngineTest (7: delegates to a dedicated ClientDataFlowInjector without coupling MQTT into the engine — BL-246; no schema registered; publication via the generic engine; validation holdout; -1 for other dataset types; publishes to registered clientIds, not synthetic 1..N — BL-207) · DatasetDeploymentServiceTest (6: deploy/extraction, lookup by id, error for unknown id, listing per experiment, failure without dataset-schema.json, directory isolation across uploads — BL-089)
federated-experiment-sdk ExperimentRuntimePathsTest (extended: deployedDatasetsRoot/deployedDatasetDirectory, default fallback and argument validation — BL-089)
research/experiments/exp-mnist-digits-001 MnistMqttSimulationAdapterTest/MnistFederatedSessionTest (pre-existing, behavior preserved after the migration)

Mandatory rules

  • New tabular CSV dataset with a header: do not create a {Dataset}MqttSimulationAdapter from scratch — configure a CsvTabularDatasetSchema and use CsvDatasetSimulationSource directly (or via SimulationDataPackagePlayer, if coming from a .hdtsim.zip). Only write a dataset-specific adapter if the dataset needs its own synthetic fallback or is not tabular.
  • If the CSV already identifies the client per row, use partitionStrategy="by_existing_column" plus clientIdColumn. If it does not, use sequential_chunk/random_shuffle_seed — never invent a new partitioning algorithm outside DatasetPartitioner.
  • stratified_by_label is not supported by the generic engine — datasets that need this still require a dataset-specific Java adapter with external grouping.
  • Train/test/validation segmentation is always optional and additive via extraParameters — never add new fields to DatasetSimulationConfig/SimulationSession for this; prefer the same well-known-keys-in-a-free-form-map pattern already validated in this component.
  • SimulationDataPackageWriter/Reader remain the only valid way to generate/read a .hdtsim.zip — dataset-schema.json is just another file in the same staging directory, requiring no special handling.
  • POST /datasets/deploy (BL-089) rejects packages without dataset-schema.json — the dataset-execution API is intentionally generic and must not accept packages that require a dataset-specific Java adapter.
  • POST /experiments/{id}/run-with-dataset never modifies AdminMonitoringService.runRounds or SimulationRoundEngine — it only orchestrates them in sequence (publishes the round via DatasetSimulationSource.simulateRound, then calls runRounds with numRounds=1). Any new way of triggering execution from a dataset must follow the same composition principle instead of coupling new responsibilities to those two components.