Component: Generic Dataset Simulation Engine (BL-087/BL-088/BL-089)¶
Responsibility¶
Before this delivery, each dataset (MNIST, MIMIC, ECG5000, HAR) needed its own
{Dataset}MqttSimulationAdapter that decoded its raw CSV, partitioned the rows
across clients, and built the in-memory CDM repository — Java code duplicated per dataset,
even when the input format was a common tabular CSV with a header. In addition,
SimulationDataPackageDescriptor (BL-074) already declared numClients/numRounds/
partitionStrategy in the .hdtsim.zip package, but no code read those fields back to
configure an executable simulation — they were just documentation read by the researcher.
This component solves both problems with a dataset-agnostic simulation engine:
given a tabular CSV with a header and a declarative description of its columns
(CsvTabularDatasetSchema), the engine decodes, partitions across clients, and publishes over MQTT with
no dataset-specific Java code at all. DataComponent and the federated client never know that
the generic engine exists — they continue to receive data exclusively via
DataComponent.subscribeToMqttSource/materializeDatasetFromStore/loadLocalCdmRepository
(BL-084/BL-085), with no contract change on that end.
Implementation¶
| Item | Value |
|---|---|
| Schema/descriptor module | modules/federated-experiment-packager |
| Simulation engine module | modules/federated-simulator |
| Schema package | br.ufma.lsdi.packager.simulation |
| Engine package | br.ufma.lsdi.simulator |
| Main classes | CsvTabularDatasetSchema, DatasetSimulationConfig, DatasetSimulationSource (interface), CsvDatasetSimulationSource (reference implementation), SimulationDataPackagePlayer |
| Web monitor integration | SimulationRoundEngine (datasetType="csv") — see 01-admin-api-monitor-web.md |
| Migrated pilot | research/experiments/exp-mnist-digits-001 (MnistMqttSimulationAdapter) |
| CLI/script | SimulationDataPackagerCli (optional 14-argument mode), scripts/package-simulation-data.sh (WITH_CSV_SCHEMA=1) |
| Deployment/execution HTTP API (BL-089) | DatasetDeploymentService + routes in AdminWebServer — see §10 |
ADR-004 preserved:
federated-simulatornow depends onfederated-experiment-packager(standalone, depends only onfederated-common) to reuseDatasetPartitionerand the new schema — this does not create a cycle withfederated-rest-api(the ADR-004 rule specifically forbidssimulator → admin-api, and is never forbidden in the opposite direction or for other modules).
1. CsvTabularDatasetSchema — what the CSV columns are¶
Immutable record (federated-experiment-packager) that describes the format of a tabular CSV
dataset, with no knowledge of its content:
public record CsvTabularDatasetSchema(
String targetColumn, // target column (label/target) — required
String featureColumnsSpec, // feature columns — required
String clientIdColumn, // column identifying the client — optional
String recordIdColumn // column identifying the record — optional
)
targetColumnandfeatureColumnsSpecare required — blank throwsFederatedException(ErrorCode.EXPERIMENT_PACKAGE_INVALID, ...).clientIdColumn/recordIdColumnthat are null or blank become an empty string (""); their presence is tested viahasClientIdColumn()/hasRecordIdColumn().
featureColumnsSpec notation¶
A comma-separated list and/or the compact range notation prefixSTART..prefixEND
(the same notation already used informally in HAR/ECG/MIMIC/MNIST manifest.yaml files, now
formalized and parseable via CsvTabularDatasetSchema.featureColumns()):
new CsvTabularDatasetSchema("label", "pixel_0..pixel_783", "client_id", "record_id")
.featureColumns();
// → ["pixel_0", "pixel_1", ..., "pixel_783"] (784 columns, no zero-padding)
new CsvTabularDatasetSchema("classe", "t001..t140", "", "")
.featureColumns();
// → ["t001", "t002", ..., "t140"] (zero-padding of width 3, preserved from the starting token)
new CsvTabularDatasetSchema("label", "subject_id,pixel_0..pixel_2,extra", "", "")
.featureColumns();
// → ["subject_id", "pixel_0", "pixel_1", "pixel_2", "extra"] (list and range combined)
The numeric width (zero-padding) is always inferred from the token to the left of ..; the two sides of the
range must share the same textual prefix, otherwise featureColumns() throws
IllegalArgumentException.
Persistence (dataset-schema.json)¶
Optionally written inside the .hdtsim.zip package, alongside
simulation-descriptor.json (SimulationDataPackageDescriptor, BL-074). Hand-rolled JSON (the same
style used throughout the rest of the project — no external library). SimulationDataPackageWriter/Reader
iterate generically over all files in the package — no change was needed in them to
support this new optional file.
2. Client identification — existing column or synthetic partitioning¶
This is the central point for anyone who needs to separate each client's data: the engine supports
two complementary strategies, chosen via DatasetSimulationConfig.partitionStrategy().
(a) Identify the client field in the dataset itself — by_existing_column¶
Use this when the CSV already has a column identifying the client (e.g., subject_id, client_id,
patient_id). Configure CsvTabularDatasetSchema.clientIdColumn with that column's name:
When reading the CSV, each row carries the raw value of that column (CsvRow.clientId()). During
publication, CsvDatasetSimulationSource groups the rows by that column's value:
for each row i in the dataset:
clientId = row.clientId()
if clientId is null/empty OR not among the configured clientIds:
clientId = clientIds[i % totalClients] // round-robin fallback (see below)
assign the row to that clientId
Built-in automatic fallback: if a specific row has no client value (empty cell)
or the value does not match any clientId configured in
DatasetSimulationConfig.clientIds(), that individual row is assigned a synthetic client via
round-robin — not every row needs to have the column filled in; rows with and without
client identification can coexist in the same CSV. This is exactly the same algorithm already
used by MnistMqttSimulationAdapter since BL-084, now generalized.
(b) Fully synthetic partitioning — sequential_chunk / random_shuffle_seed¶
Use this when the CSV has no client column at all — the data from a single file (e.g., a
sensor CSV with no device identification) must be artificially divided among N
simulated clients. In this case, any value in clientIdColumn is entirely ignored — the
partitioning ignores row content and uses only the total count, delegating to
DatasetPartitioner (federated-experiment-packager, existing since BL-074, but never
consumed by any code before BL-088):
DatasetPartitioner.partitionClients(
"sequential_chunk", // or "random_shuffle_seed"
totalRows, clientIds, partitionSeed);
// → Map<clientId, List<Integer>> (row indices, no overlap, covering all rows)
sequential_chunk: contiguous blocks in the file's original order (client 1 = rows[0, chunkSize), client 2 =[chunkSize, 2*chunkSize), etc.; the last client absorbs the remainder when the division is not exact).random_shuffle_seed: shuffles the indices deterministically (Random(partitionSeed)) before slicing into contiguous blocks — same deterministic coverage, different order.
Choosing between the two¶
| Situation | partitionStrategy |
clientIdColumn |
|---|---|---|
CSV already identifies the client per row (e.g., subject_id) |
by_existing_column |
column name |
| CSV has no client column — split artificially | sequential_chunk or random_shuffle_seed |
"" (ignored) |
| CSV has the column, but some rows have no value | by_existing_column |
column name (round-robin fallback covers rows without a value) |
stratified_by_label (one of the 4 strategies documented in SimulationDataPackageDescriptor,
see 14-iot-replay-simulator.md and the BL-074 descriptor) is not
supported by the generic engine in this delivery — it requires external grouping by class label;
DatasetSimulationConfig explicitly rejects this strategy with a FederatedException. Datasets that
require stratified partitioning still require a dataset-specific Java adapter.
3. DatasetSimulationConfig / DatasetSimulationSource — the data-component interface¶
DatasetSimulationConfig is the immutable, validated record that replaces hardcoded per-adapter
constants (e.g., MNIST's old DEFAULT_CLIENT_IDS):
public record DatasetSimulationConfig(
String experimentId,
List<String> clientIds, // no duplicates, no blank entries
int totalRounds, // > 0
String partitionStrategy, // by_existing_column | sequential_chunk | random_shuffle_seed
long partitionSeed, // ignored outside random_shuffle_seed
String senderIdPrefix, // never null (empty by default)
Map<String, String> extraParameters // free-form — see train/test/validation segmentation (§4)
)
DatasetSimulationSource is the interface that any simulation engine must implement:
public interface DatasetSimulationSource {
List<MqttDatasetReplaySimulator.ReplaySummary> simulateAll(
DatasetSimulationConfig config, MqttGateway mqttGateway, HmacSignatureService signatureService);
List<MqttDatasetReplaySimulator.ReplaySummary> simulateRound(
DatasetSimulationConfig config, int roundNumber, MqttGateway mqttGateway,
HmacSignatureService signatureService);
}
It receives a concrete HmacSignatureService (not the SignatureService interface) because the
simulation always needs to register a deterministic identity per client (registerIdentity) before
signing — the same pattern already used by the dataset-specific adapters.
4. CsvDatasetSimulationSource — reference implementation¶
Decodes rows (from a real CSV file or from an in-memory CsvRow list — an extension
point for a dataset-specific synthetic fallback, see §6), partitions per §2,
builds the in-memory CDM repository, and delegates all MQTT publication to
MqttDatasetReplaySimulator.replayRepository(...) — no new transport, the same signed
envelope (MqttEnvelope/MqttGateway) used throughout the rest of the project.
CsvTabularDatasetSchema schema = new CsvTabularDatasetSchema(
"label", "pixel_0..pixel_783", "client_id", "record_id");
CsvDatasetSimulationSource source = CsvDatasetSimulationSource.fromCsvFile(schema, csvFile);
DatasetSimulationConfig config = new DatasetSimulationConfig(
"exp-meu-experimento-001", List.of("1", "2", "3"), 5,
"by_existing_column", 0L, "meu-sim-", Map.of());
List<MqttDatasetReplaySimulator.ReplaySummary> resumo =
source.simulateAll(config, mqttGateway, signatureService);
The published CDM vocabulary (entity/event) is deliberately generic
(csv_tabular_record/csv_record_labeled/csv) — the observed-property codes
(what actually matters for reconstructing features during training) always come from
schema.featureColumns()/schema.targetColumn(), preserving names like pixel_0..pixel_783
intact between the source CSV and what reaches the client over MQTT.
simulateAll vs simulateRound¶
simulateAllpublishes, for each client, all the rows that belong to it — covers the entire dataset in a single call.simulateRound(config, roundNumber, ...)publishes only the slice for roundroundNumber(1-indexed) out ofconfig.totalRounds(), viaDatasetPartitioner.roundSlice(the same slicing already used byHarMqttReplayAdapter/EcgMqttReplayAdapter:chunkSize = max(1, total/totalRounds), the last round absorbs the remainder).
5. Train/test/validation segmentation (BL-088)¶
Optional support for splitting the dataset into three subsets by a configurable ratio, without
changing the already-tested signature of DatasetSimulationConfig — the ratios are passed via
extraParameters, well-known keys read only by CsvDatasetSimulationSource:
| Key | Required? | Description |
|---|---|---|
trainRatio |
Yes, if any of the 3 is present | Fraction of rows for training, [0.0, 1.0] |
testRatio |
Yes, if any of the 3 is present | Fraction of rows for testing, [0.0, 1.0] |
validationRatio |
Yes, if any of the 3 is present | Fraction of rows for validation, [0.0, 1.0] |
splitSeed |
No | Shuffle seed; uses partitionSeed if absent |
Map<String, String> extra = Map.of(
"trainRatio", "0.7", "testRatio", "0.15", "validationRatio", "0.15");
DatasetSimulationConfig config = new DatasetSimulationConfig(
"exp-001", List.of("1", "2"), 5, "by_existing_column", 0L, "sim-", extra);
Rules:
- The 3 keys must appear together. If any is missing, no segmentation is applied — behavior before BL-088 is preserved by default (all rows go to the clients).
- The 3 ratios must sum to
1.0(tolerance1e-6); otherwise,CsvDatasetSimulationSourcethrowsIllegalArgumentException. - Explicit user decision: only
validationis retained as a central holdout — it is never published over MQTT.trainandtestare combined and distributed normally to the clients (both "go to the clients"). - The validation subset is accessible via
source.validationRows(config)— deterministic (the sameconfig/seed always produces the same result), useful for central evaluation by the researcher outside the MQTT flow.
List<CsvDatasetSimulationSource.CsvRow> holdout = source.validationRows(config);
// validation rows — never published, available for central evaluation
6. SimulationDataPackagePlayer — from a .hdtsim.zip package to an executable simulation¶
Closes the BL-074 gap: before this class, the numClients/numRounds/
partitionStrategy fields of SimulationDataPackageDescriptor were only documentation. The player reads the
descriptor plus the dataset-schema.json from an already-extracted directory (via SimulationDataPackageReader)
and assembles a ready-to-run simulation:
List<MqttDatasetReplaySimulator.ReplaySummary> resumo = SimulationDataPackagePlayer.simulateAll(
extractedDirectory, descriptor, mqttGateway, signatureService, "player-sim-");
// default numeric clientIds "1".."numClients" when not explicitly provided
load(...) throws FederatedException(ErrorCode.EXPERIMENT_PACKAGE_INVALID, ...) if
dataset-schema.json does not exist in the extracted directory — required so the generic engine
knows how to decode the package's CSV. Datasets that require dataset-specific decoding
(non-tabular formats, or their own synthetic fallback) continue to use their own
{Dataset}MqttSimulationAdapter instead of this player.
7. Packaging CLI and script¶
SimulationDataPackagerCli optionally accepts 4 more arguments (14 total) to also
write dataset-schema.json to the staging directory:
java -cp ... br.ufma.lsdi.packager.simulation.SimulationDataPackagerCli \
<staging> <output-zip> <experimentId> <datasetEntry> <datasetDescription> <partitionStrategy> \
<partitionDescription> <numClients> <numRounds> <samplesPerClient> \
<targetColumn> <featureColumnsSpec> <clientIdColumn> <recordIdColumn> # optional
scripts/package-simulation-data.sh exposes this interactively via WITH_CSV_SCHEMA=1:
WITH_CSV_SCHEMA=1 sh scripts/package-simulation-data.sh exp-meu-experimento-001 \
dataset.csv 3 5 by_existing_column
# Interactively prompts for: targetColumn, featureColumnsSpec, clientIdColumn, recordIdColumn
Without WITH_CSV_SCHEMA=1, the package remains valid (legacy mode, descriptor only from BL-074) but requires
a dataset-specific Java adapter to be simulated.
8. Web monitor integration¶
The monitor's round-based simulation session ("▶ Session", SimulationRoundEngine/
AdminWebServer — see 01-admin-api-monitor-web.md and
14-iot-replay-simulator.md) gained datasetType="csv", alongside
har/ecg (which continue to use their untouched dataset-specific adapters). Since SimulationSession
is a record with many production/test call sites, the CSV schema and the
train/test/validation segmentation were not added to it — they are registered separately, before
starting the session:
roundEngine.registerCsvSchema(experimentId, schema); // required for datasetType=csv
roundEngine.registerCsvSplitRatios(experimentId, splitParams); // optional
roundEngine.registerClientIds(experimentId, expectedClientIds); // BL-207 — see below
int holdout = roundEngine.csvValidationRowCount(session); // -1 if not applicable
BL-207 —
registerClientIdsis not optional in practice. Without it the CSV replay generates synthetic ids"1".."maxClients". Since the publishedsenderIdiscsv-sim-{clientId}andMqttTimeSeriesIngestoronly accepts its own client's sender, publishing to synthetic ids when the registered clients carry different ids (saysubject-07) means publishing to clients nobody is listening as — the real clients end up with an empty dataset and no error anywhere.AdminWebServernow calls it at session start withAdminMonitoringService.expectedClientIds(...).POST /experiments/{id}/run-with-datasetalready resolved client ids correctly; only the simulation session did not. HAR/ECG do not use this registry — their client ids come from the dataset's own subjects, which is the correct source for them.
The session-creation form (/experiments/{id}/sim-session) exposes the
"CSV — Generic Engine" option with conditional fields (targetColumn/featureColumnsSpec/
clientIdColumn/recordIdColumn plus the 3 split ratios); a csv-schema-required banner covers
missing required fields; the active session page shows the retained validation
holdout count, when applicable.
9. Migrated pilot — MnistMqttSimulationAdapter¶
MnistMqttSimulationAdapter.simulate(datasetRoot, clientIds, mqttGateway, signatureService,
senderIdPrefix) kept its public signature unchanged, but now internally delegates to the
generic engine. The only MNIST-specific code left in the adapter is:
- Locating the real CSV (
mnist_train.csv/dataset.csv); - The deterministic synthetic-fallback formula (10 images per client, one per digit 0-9) — always dataset-specific, never generic.
List<CsvDatasetSimulationSource.CsvRow> rows =
CsvDatasetSimulationSource.readRowsFromCsv(SCHEMA, resolveDatasetFile(resolvedRoot));
if (rows.isEmpty()) {
rows = syntheticRows(clientIds); // the only MNIST-specific piece
}
CsvDatasetSimulationSource source = new CsvDatasetSimulationSource(SCHEMA, rows);
MIMIC/ECG/HAR were not migrated in this delivery (the same incremental-scope decision as BL-084/085) — direct candidates, since they already have CSV tabular loaders structurally equivalent to MNIST's.
10. Dataset deployment/execution HTTP API (BL-089)¶
Closes the final gap in the cycle: until now, publishing data from a .hdtsim.zip package over MQTT
required Java code calling SimulationDataPackagePlayer directly, or the manual flow of the
round-based simulation session (§8). This API exposes upload and execution over HTTP, for
any experiment with a dataset package declaring dataset-schema.json — it is not
MNIST-specific.
DatasetDeploymentService — deployment (POST /datasets/deploy)¶
Mirrors ExperimentDeploymentService/POST /experiments/deploy (ADR-005), but for data, not
code: it extracts the zip with the same protections as SimulationDataPackageReader (zip slip,
zip bomb, SHA256SUMS checksum) and loads and executes no code at all. Unlike experiment
deployment, it does not deduplicate by content hash — each upload receives a datasetPackageId
(UUID) and an isolated directory under ExperimentRuntimePaths.deployedDatasetDirectory(experimentId,
datasetPackageId) (root configurable via HDT_AF_DEPLOYED_DATASETS_ROOT/
DEPLOYED_DATASETS_ROOT, default var/deployed-datasets/ — legacy fallback target/deployed-datasets/), allowing resubmissions
(new versions of the data), all traceable and available for execution.
Immediately rejects (FederatedException(EXPERIMENT_PACKAGE_INVALID)) packages without
dataset-schema.json — a fast, actionable failure at upload time, instead of only at execution time.
Routes:
| Route | Method | Description |
|---|---|---|
/datasets/deploy |
GET |
Upload form |
/datasets/deploy |
POST (multipart) |
Deploys the package; redirects to /experiments/{id}/datasets |
/experiments/{id}/datasets |
GET |
Lists the packages deployed for the experiment, with an execution form per package |
/experiments/{id}/run-with-dataset |
POST |
Executes rounds from a deployed package (see below) |
/api/experiments/{id}/datasets |
GET |
JSON of the deployed packages |
POST /experiments/{id}/run-with-dataset — dedicated execution¶
A new route, independent of POST /experiments/{id}/run-rounds — it does not change that method or the
round-based simulation session flow (§8); it merely orchestrates them in sequence:
for round in 1..numRounds:
SimulationDataPackagePlayer.load(...).simulateRound(config, round, mqttGateway, signatureService)
→ publishes the round's slice over MQTT (the same signed envelope as always)
AdminMonitoringService.runRounds(experimentId, contractVersion, 1, fineTune only on the last round)
→ triggers exactly one federated round (session.executeRound)
Form parameters (urlencoded, not multipart — the package was already uploaded via /datasets/deploy):
| Field | Required | Description |
|---|---|---|
datasetPackageId |
Yes | Id returned by /datasets/deploy |
contractVersion |
No (default 1.0.0) |
Contract version to execute |
numRounds |
No (default = the descriptor's numRounds) |
Rounds to execute, capped at the total declared in the package |
fineTune |
No | If checked, runs +5 fine-tuning rounds after the last requested round |
clientIds is resolved automatically from
AdminMonitoringService.expectedClientIds(experimentId, contractVersion) (clients already
registered on the experiment); if none are registered, it falls back to "1".."numClients" from the
package's descriptor.
Session prerequisite: before publishing the first round, the route calls
runRounds(experimentId, contractVersion, 0, false) — this creates the experiment session (and, with
it, the clients' MQTT subscription) without executing any round, ensuring the clients are already
subscribed when the first data point is published. Requires a MqttGateway configured in the
monitor (the same requirement as datasetType="csv" in §8) — without it, the route throws
IllegalStateException.
Not verified end-to-end in this delivery: the full integration (upload → MQTT publish → round execution → aggregation) depends on Maven infrastructure unavailable in this development environment (see the verification note in
backlog.md, BL-089). The logic was reviewed statically, line by line, against the real signatures ofSimulationDataPackagePlayer,AdminMonitoringService.runRounds, andDatasetSimulationSource, but runningsh scripts/run-tests.sh --module federated-rest-apiand a manual end-to-end test is recommended before production use.
Tests¶
| Module | Coverage |
|---|---|
federated-experiment-packager |
CsvTabularDatasetSchemaTest (9: validation, range expansion with/without padding, list+range mix, JSON round-trip) · SimulationDataPackagerCliTest (2: legacy mode vs. mode with schema) |
federated-simulator |
DatasetSimulationConfigTest (11: validation of each field, defaults, immutability) · CsvDatasetSimulationSourceTest (per-record counts, per-client identity, round-robin fallback, sequential_chunk, simulateRound/slicing, invalid roundNumber, TrainTestValidationSplitTests with 5 cases) · SimulationDataPackagePlayerTest (3: end-to-end simulation from an extracted package, error without dataset-schema.json, buildConfig defaults) |
federated-rest-api |
SimulationRoundEngineTest (7: delegates to a dedicated ClientDataFlowInjector without coupling MQTT into the engine — BL-246; no schema registered; publication via the generic engine; validation holdout; -1 for other dataset types; publishes to registered clientIds, not synthetic 1..N — BL-207) · DatasetDeploymentServiceTest (6: deploy/extraction, lookup by id, error for unknown id, listing per experiment, failure without dataset-schema.json, directory isolation across uploads — BL-089) |
federated-experiment-sdk |
ExperimentRuntimePathsTest (extended: deployedDatasetsRoot/deployedDatasetDirectory, default fallback and argument validation — BL-089) |
research/experiments/exp-mnist-digits-001 |
MnistMqttSimulationAdapterTest/MnistFederatedSessionTest (pre-existing, behavior preserved after the migration) |
Mandatory rules¶
- New tabular CSV dataset with a header: do not create a
{Dataset}MqttSimulationAdapterfrom scratch — configure aCsvTabularDatasetSchemaand useCsvDatasetSimulationSourcedirectly (or viaSimulationDataPackagePlayer, if coming from a.hdtsim.zip). Only write a dataset-specific adapter if the dataset needs its own synthetic fallback or is not tabular. - If the CSV already identifies the client per row, use
partitionStrategy="by_existing_column"plusclientIdColumn. If it does not, usesequential_chunk/random_shuffle_seed— never invent a new partitioning algorithm outsideDatasetPartitioner. stratified_by_labelis not supported by the generic engine — datasets that need this still require a dataset-specific Java adapter with external grouping.- Train/test/validation segmentation is always optional and additive via
extraParameters— never add new fields toDatasetSimulationConfig/SimulationSessionfor this; prefer the same well-known-keys-in-a-free-form-map pattern already validated in this component. SimulationDataPackageWriter/Readerremain the only valid way to generate/read a.hdtsim.zip—dataset-schema.jsonis just another file in the same staging directory, requiring no special handling.POST /datasets/deploy(BL-089) rejects packages withoutdataset-schema.json— the dataset-execution API is intentionally generic and must not accept packages that require a dataset-specific Java adapter.POST /experiments/{id}/run-with-datasetnever modifiesAdminMonitoringService.runRoundsorSimulationRoundEngine— it only orchestrates them in sequence (publishes the round viaDatasetSimulationSource.simulateRound, then callsrunRoundswithnumRounds=1). Any new way of triggering execution from a dataset must follow the same composition principle instead of coupling new responsibilities to those two components.