Ir para o conteúdo

Component: Experiment Packaging and Dynamic Deployment

Responsibility

Before this delivery (ADR-005, BL-056 through BL-059), an experiment only existed if its Java source code was statically included in federated-rest-api's pom.xml (via build-helper-maven-plugin) and was only executable if AdminExperimentExecutor had a hardcoded case for its experimentId. There was no distributable artifact and no upload — adding an experiment required editing and recompiling federated-rest-api.

This component defines the system's single packaging module, federated-experiment-packager, and the service that dynamically deploys experiment packages to the web monitor (ExperimentDeploymentService, in federated-rest-api). The module owns the experiment, simulation data and signed Android client package formats, including their descriptors, signatures, readers/writers and CLIs. There are no separate client packaging API or client packager modules.


Implementation

Item Value
Format module modules/federated-experiment-packager
Java packages br.ufma.lsdi.packager, br.ufma.lsdi.packager.simulation, br.ufma.lsdi.packager.client
Main classes ExperimentPackage*, SimulationDataPackage*, ClientPackage* descriptors/readers/writers (ClientPackageSignature) and their CLIs
Deploy service br.ufma.lsdi.admin.ExperimentDeploymentService (federated-rest-api module)
Upload parser br.ufma.lsdi.admin.MultipartFormParser
Process isolation (BL-236) br.ufma.lsdi.admin.isolated.* (federated-rest-api module)
Packaging script scripts/package-experiment.sh {experimentId}
Deploy endpoint GET/POST /experiments/deploy (see docs/components/01-admin-api-monitor-web.md)

Package format — ExperimentPackage v1 ({experimentId}.hdtpkg.zip)

package-descriptor.json      # machine-readable descriptor (Jackson JSON)
manifest.yaml                # copy of the human-readable manifest (documentation/provenance)
specification.md
research.xml                 # optional, if it exists
docker.env.example
code/{experimentId}.jar      # compiled experiment classes (no fat-jar; runs on top of
                              # the classpath already present in admin-api: sdk, common, client, aggregator...)
SHA256SUMS                   # sha256 hash of each file above, in a deterministic order (SHA256SUMS last)

package-descriptor.json:

{
  "packageFormatVersion": 1,
  "experimentId": "exp-ecg5000-anomaly-001",
  "contractVersion": "1.0.0",
  "sessionFactoryClass": "br.ufma.lsdi.research.ecg.EcgSessionFactory",
  "codeJarEntry": "code/exp-ecg5000-anomaly-001.jar"
}

manifest.yaml/specification.md/research.xml remain documentation/provenance (their hash is recorded in SHA256SUMS, they are not parsed by code) — they do not duplicate the source of truth.

ExperimentPackageDescriptor

Record serialized/deserialized via Jackson ObjectMapper (toJson / fromJson, just like ClientPackageDescriptor). Note that PackageZipSupport.verifyChecksums tolerates packages without SHA256SUMS for legacy compatibility, whereas ClientPackageReader enforces SHA256SUMS presence. ExperimentPackageDescriptor rejects:

  • a missing packageFormatVersion or one other than 1;
  • a blank or missing experimentId, sessionFactoryClass, or codeJarEntry.

All rejections throw FederatedException(ErrorCode.EXPERIMENT_PACKAGE_INVALID, ...).

ExperimentPackageWriter

Assembles the zip from a staging directory that already contains package-descriptor.json plus the remaining files. Rules:

  • Entries in deterministic alphabetical order; SHA256SUMS is always the last entry.
  • SHA256SUMS uses Hashing.sha256(Path) (streaming, without loading large files into memory) — line format: <hash> <relative path>.
  • Fails if code/{id}.jar does not contain the .class entry corresponding to the declared sessionFactoryClass in the descriptor (zip index check via JarFile.getEntry(...), does not load the class) — catches the most common error before deployment.

ExperimentPackageReader

Extracts to a destination directory with the same protections as LocalArtifactExperimentService.resolveSecure():

  • Zip slip: each entry is resolved and normalized against the destination directory; entries that escape (../) are rejected before any write happens.
  • Zip bomb: each entry is copied with a configurable limit on decompressed bytes (ExperimentPackageReader(long maxEntryUncompressedBytes), default 200 MB) — the limit is enforced during the copy (actual bytes read), not trusting ZipEntry metadata.
  • Integrity: after extraction, it recalculates the hash of every file listed in SHA256SUMS and compares them — a mismatch throws FederatedException(ErrorCode.PACKAGE_HASH_MISMATCH, ...).
  • Format version: an unknown packageFormatVersion is rejected by ExperimentPackageDescriptor.fromJson(...) itself when parsing the extracted descriptor.

scripts/package-experiment.sh {experimentId}

sh scripts/package-experiment.sh exp-ecg5000-anomaly-001
# → target/packages/exp-ecg5000-anomaly-001.hdtpkg.zip

Optional flow with direct submission to the platform:

sh scripts/package-experiment.sh exp-ecg5000-anomaly-001 --submit
sh scripts/package-experiment.sh exp-ecg5000-anomaly-001 --submit --client-ids 1,2,3
sh scripts/package-experiment.sh exp-ecg5000-anomaly-001 --submit --platform-url http://127.0.0.1:8088

When --submit is used, the script sends multipart/form-data to POST /experiments/deploy with the package and clientIds fields, reusing the same endpoint as the web monitor. The raw HTTP response is saved to target/packages/{experimentId}.deploy.response.

Flow: mvn -pl research/experiments/{id} -am package -DskipTests (compiles the experiment's standalone Maven module — prerequisite: its own pom.xml, artifactId equal to the experimentId, see docs/components/13-research-experiments.md) → builds a staging directory by copying manifest.yaml/specification.md/research.xml/docker.env.example from the experiment and the newly compiled jar → writes package-descriptor.json → invokes ExperimentPackagerCli (a simple main class, java -cp ... ExperimentPackagerCli <staging> <output-zip>, without exec-maven-plugin).

The experimentId → sessionFactoryClass/contractVersion mapping is a case table inside the script itself — today it covers ECG5000, MIMIC v3.1, MNIST, and ergonomic-risk experiment packages. HAR/DT requires adding an entry here and its own pom.xml for the experiment, following the same pattern.


Dynamic deploy — ExperimentDeploymentService (ADR-005)

ExperimentDeploymentService service = new ExperimentDeploymentService(executor, monitoringService);
ExperimentContract contract = service.deploy(packageZipBytes, List.of("1", "2", "3"));

Flow of deploy(byte[] packageZipBytes, List<String> clientIds):

  1. Writes the bytes to a temporary file and extracts them via ExperimentPackageReader into a temporary staging directory (zip slip/bomb/hash/version checks already covered — see above).
  2. Computes a shortHash (12 characters of the SHA-256 of the zip's bytes) and moves the extracted content to ExperimentRuntimePaths.deployedExperimentDirectory(experimentId, contractVersion, shortHash) — a directory versioned per deploy, preventing a redeploy from overwriting a jar in use by an isolated process already spawned for a running session. A redeploy of identical content (same hash) reuses the existing directory (idempotent).
  3. Validates sessionFactoryClass by launching a disposable isolated OS process (see "Process isolation (BL-236)" below) and sending it a VALIDATE command — the class is loaded and instantiated inside that separate process, never inside admin-api's own JVM, and the check confirms instanceof ExperimentSessionFactory before the process exits.
  4. Registers an IsolatedExperimentSessionFactory — never the loaded factory itself — that knows how to launch a fresh isolated process on demand for every subsequent create(...).
  5. AdminExperimentExecutor.registerFactory(experimentId, contractVersion, factory) — dynamic registration takes priority over the hardcoded legacy switch (HAR/ECG/DT continue working unchanged; the dynamic registry is consulted first in createSession/execute). Since BL-077, the registry and dynamic sessions are keyed by (experimentId, contractVersion), allowing two versions of the same experiment to coexist in the same process.
  6. Creates an ExperimentContract via ExperimentContractFactory.buildDraft(...).withStatus(ContractStatus.ACTIVE) — it is born ACTIVE (not DRAFT) because the goal is to deploy and train immediately; the metadata fields (coordinator, objective, digital twin type, task type) use generic values since package-descriptor.json does not carry those fields.
  7. AdminMonitoringService.registerExperiment(contract, clientIds) — the same path used by AdminApplicationBootstrap.registerDefaultExperiments().

Operational versioning

  • The directory extracted on disk remains isolated by contractVersion + shortHash.
  • The monitor now keeps a separate stateful session for each (experimentId, contractVersion).
  • A redeploy of version 2.0.0 does not automatically remove the already-running session of version 1.0.0.
  • Session reset (POST /experiments/{id}/reset-session?version=...) affects only the specified version.
  • A call with no explicit version in the executor (execute(experimentId)) uses the most recently registered dynamic version for that experimentId, preserving compatibility with legacy flows.

Process isolation (BL-236)

Before this delivery, sessionFactoryClass was loaded and executed inside admin-api's own JVM via URLClassLoader — the risk was documented and explicitly accepted (see "Accepted risk" below, previous revision). This delivery replaces that with genuine OS-level process isolation:

br.ufma.lsdi.admin.isolated.IsolatedExperimentRunnerApp   // entry point of the isolated process
br.ufma.lsdi.admin.isolated.IsolatedExperimentRunner      // dispatch logic (testable without a process)
br.ufma.lsdi.admin.isolated.SubprocessIsolatedProcessLauncher   // production launcher (real java -cp ... subprocess)
br.ufma.lsdi.admin.isolated.IsolatedExperimentSession           // ExperimentSession proxy over the child process
br.ufma.lsdi.admin.isolated.IsolatedExperimentSessionFactory    // ExperimentSessionFactory registered instead of the raw factory
  • Protocol: one JSON line per command on the child's stdin, one JSON line per response on its stdout (IsolatedExperimentRequest/IsolatedExperimentResponse, encoded via the shared br.ufma.lsdi.common.util.JsonCodec). Commands: VALIDATE (load + instantiate only, no side effect — used by ExperimentDeploymentService.loadFactory), LOAD (also calls create(...), the resulting session is retained inside the child process), EXECUTE_ROUND, TRIGGER_EXTERNAL.
  • stderr is inherited from the parent (ProcessBuilder.Redirect.INHERIT) — library logging never shares the pipe used by the protocol on stdout, and never risks filling an undrained buffer and deadlocking the child.
  • Classpath: the child process is launched with the same java binary (ProcessHandle.current().info().command()) and the same java.class.path as admin-api itself — the deployed package is not a fat-jar, it expects federated-common/-client/ -aggregator/etc. to already be on the classpath, exactly as the old URLClassLoader used AdminExperimentExecutor.class.getClassLoader() as its parent.
  • Transparency to the rest of the system: AdminExperimentExecutor needed zero changes — it already depended only on the ExperimentSessionFactory/ExperimentSession interfaces. Swapping "factory loaded in-process" for "factory that knows how to launch an isolated process" is entirely invisible to its 15+ existing tests.
  • Verified for real, not just at the interface level: IsolatedExperimentRunnerAppIntegrationTest (@Tag("integration")) spawns an actual java child process (own PID, own JVM) with SubprocessIsolatedProcessLauncher, loads a fixture jar in it, executes a round, and confirms destroy() actually terminates the OS process (ProcessHandle.current().children() before/after) — no Docker dependency, ~1s total.
  • What is still out of scope: the HTTP deploy endpoint itself still has no authentication of its own (same trust model as every other admin-api route today — see "Accepted risk"); resource limits (CPU/memory/time) on the isolated process; and container-level isolation (Docker per deploy) as a stronger future option if the trust model around port 8088 ever changes.

MultipartFormParser

A minimal multipart/form-data (RFC 7578) parser with no external dependencies — used only by the deploy endpoint, since the body contains a binary file (the existing AdminWebServer.form(...) assumes application/x-www-form-urlencoded and would corrupt the upload). Returns text fields (Map<String,String>) and file parts (List<FilePart> with field name, file name, content-type, and bytes).

HTTP endpoint

Method Route Description
GET /experiments/deploy Upload form (package: file; clientIds: optional, CSV)
POST /experiments/deploy Receives the multipart, delegates to ExperimentDeploymentService, redirects (303) to /experiments/{id}?version={contractVersion}

The route is intercepted in AdminWebServer.handle(...) before the generic path.startsWith("/experiments/") dispatch, because that generic path (routeExperiment) already consumes the request body as a URL-encoded form — consuming the body twice, or as the wrong type, would corrupt the binary upload.


Accepted risk (explicit user/operator decision)

Risk Status
RCE via upload — anyone with access to port 8088 can upload an arbitrary jar and have it loaded and executed, with no token, no origin restriction, no mTLS Mitigated (BL-236). Uploaded code now always runs in a genuinely separate OS process (own PID, own JVM — see "Process isolation (BL-236)" above), never inside admin-api's own JVM. The endpoint itself still has no authentication — the monitor has none today for any other route either, and the operator chose to keep that trust model for deploy. Do not expose port 8088 outside a trusted network.
Zip slip during extraction Mitigated — ExperimentPackageReader normalizes and validates each entry against the destination directory before writing.
Zip bomb Mitigated — configurable limit on decompressed bytes per entry, enforced during the actual copy.
Corrupted/tampered jar in transit Mitigated — SHA256SUMS verified after extraction, before the isolated process loads any class.
Version collision under an active session Mitigated — extraction directory versioned by contractVersion + shortHash; an identical redeploy is idempotent, a redeploy with new content uses a new directory.
Regression in the hardcoded legacy HAR/ECG/DT path Mitigated — the legacy switch in AdminExperimentExecutor was not removed; the dynamic registry is only consulted first, never replacing the fallback.
Isolated process left unbounded (CPU/memory/wall-clock) Not yet mitigated — the child process inherits no resource limits from the launcher; a malicious/buggy package can still exhaust host resources. Documented as a follow-up, not blocking this delivery (it removes RCE against the monitor's own process/data, which was the accepted risk here).

Explicit non-goal (unchanged): JVM sandboxing. SecurityManager has been deprecated since JEP 411 (Java 17+) and is not a viable isolation option. Process isolation (this delivery) already gives the meaningful guarantee — the deployed code can no longer read/corrupt admin-api's own memory, credentials, or other experiments' in-process state; it runs as an unprivileged sibling process instead. Container-level isolation (e.g., a Docker sandbox per deploy, with real resource limits) remains a possible stronger future step if the trust model around port 8088 ever changes.


Manual verification

sh scripts/package-experiment.sh exp-ecg5000-anomaly-001
sh scripts/run-admin-web.sh
# Go to http://127.0.0.1:8088/experiments/deploy, submit
# target/packages/exp-ecg5000-anomaly-001.hdtpkg.zip, confirm that the experiment appears
# on the dashboard and that a round runs successfully.

Tests

Module Coverage
federated-experiment-packager Experiment descriptor/writer/reader and simulation package tests; ClientPackageDescriptorTest, ClientPackageSignatureTest, ClientPackageWriterReaderTest, ClientPackagePackagerCliTest (client descriptor validation, Ed25519 signing, DEX/hash checks, safe archive reading/writing and CLI)
federated-rest-api MultipartFormParserTest, AdminExperimentExecutorTest (dynamic registration, override, legacy fallback), ExperimentDeploymentServiceTest (fixture packaged at runtime via JarOutputStream, deploy validated through a fake isolated channel), ExperimentDeployHttpTest (end-to-end HTTP: real AdminWebServer + real multipart), br.ufma.lsdi.admin.isolated.* (BL-236: IsolatedExperimentRunnerTest, IsolatedExperimentSessionTest, IsolatedExperimentSessionFactoryTest — fake channel, no process; IsolatedExperimentRunnerAppIntegrationTest, @Tag("integration") — real java subprocess spawned end-to-end)
research/experiments/exp-ecg5000-anomaly-001 EcgSessionFactoryTest (tag e2e: EcgSessionFactory.create(...) produces an executable session against a real MQTT broker and the ECG5000 dataset)

Mandatory rules

  • New packageable experiment: create its own pom.xml in research/experiments/{id}/ (artifactId equal to experimentId) and a {Id}SessionFactory implements ExperimentSessionFactory in the same package as the existing session.
  • Never remove the hardcoded legacy path in AdminExperimentExecutor when adding a dynamic registration for an already-supported experiment.
  • ExperimentPackageWriter/Reader are the only valid way to generate/read a .hdtpkg.zip — do not assemble zips manually for deploy.
  • Any new metadata field a dynamic ExperimentContract needs (coordinator, real objective, digital twin type) must be added to package-descriptor.json and to ExperimentPackageDescriptor, never hardcoded in ExperimentDeploymentService.

Independent Android client algorithm package (.hdtclient.zip)

The signed client package format and its CLI are part of this same federated-experiment-packager module. The Android client and REST deployment service consume the descriptor/signature classes from this module; they do not bring a second packaging library into the system.

The experiment/server package (.hdtpkg.zip) does not contain Android training code. The optional Android algorithm package is a separate artifact, versioned by (experimentId, contractVersion), and can be deployed without rebuilding or reinstalling the APK:

CLIENT_PACKAGE_SIGNING_KEY=/secure/release-key.pem \
CLIENT_PACKAGE_SIGNING_KEY_ID=release-1 \
sh scripts/package-federated-client.sh exp-mdpl-001 1.0.0 \
  federated_mlp_classifier 1 br.ufma.lsdi.client.plugin.MlpTrainingPlugin \
  path/to/classes.dex --submit

The command writes target/client-packages/{experimentId}-{contractVersion}.hdtclient.zip. --submit sends it to POST /api/client-packages/deploy; the monitor verifies the Ed25519 signature and trusted key ID before making the package available through GET /api/experiments/{experimentId}/client-package?version={contractVersion}. A different package cannot replace an already deployed package for the same contract version; publish a new contract version to update its algorithm.

The archive contains code/client-engine.dex (enforced code/*.dex entry by ClientPackageDescriptor and staged as code/client-engine.dex by ClientPackagePackagerCli), client-package-descriptor.json, client-package-signature.sig (Ed25519 signature), and SHA256SUMS. The private key remains in the packaging environment. The server is configured with HDT_AF_CLIENT_PACKAGE_TRUSTED_PUBLIC_KEY and HDT_AF_CLIENT_PACKAGE_TRUSTED_KEY_ID; the APK pins the matching public key using CLIENT_PACKAGE_TRUSTED_PUBLIC_KEY and CLIENT_PACKAGE_TRUSTED_KEY_ID.

On device, the package manager verifies the archive before using DexClassLoader. The plugin runs inside the preinstalled MDPL OnArrivalPhenotypeInferenceEngine; local request/result transport uses CDDL and never forwards training data to the remote MQTT broker. When no client package is published, Android uses its built-in LocalTrainer. This packaging/deployment path is independent of the .hdtpkg.zip experiment deployment described above; see 06-federated-client.md for the Android runtime and the API routes.