Component: Experiment Packaging and Dynamic Deployment¶
Responsibility¶
Before this delivery (ADR-005, BL-056 through BL-059), an experiment only existed if its Java source
code was statically included in federated-rest-api's pom.xml (via build-helper-maven-plugin)
and was only executable if AdminExperimentExecutor had a hardcoded case for its
experimentId. There was no distributable artifact and no upload — adding an experiment required editing
and recompiling federated-rest-api.
This component defines the system's single packaging module, federated-experiment-packager, and
the service that dynamically deploys experiment packages to the web monitor
(ExperimentDeploymentService, in federated-rest-api). The module owns the experiment, simulation
data and signed Android client package formats, including their descriptors, signatures,
readers/writers and CLIs. There are no separate client packaging API or client packager modules.
Implementation¶
| Item | Value |
|---|---|
| Format module | modules/federated-experiment-packager |
| Java packages | br.ufma.lsdi.packager, br.ufma.lsdi.packager.simulation, br.ufma.lsdi.packager.client |
| Main classes | ExperimentPackage*, SimulationDataPackage*, ClientPackage* descriptors/readers/writers (ClientPackageSignature) and their CLIs |
| Deploy service | br.ufma.lsdi.admin.ExperimentDeploymentService (federated-rest-api module) |
| Upload parser | br.ufma.lsdi.admin.MultipartFormParser |
| Process isolation (BL-236) | br.ufma.lsdi.admin.isolated.* (federated-rest-api module) |
| Packaging script | scripts/package-experiment.sh {experimentId} |
| Deploy endpoint | GET/POST /experiments/deploy (see docs/components/01-admin-api-monitor-web.md) |
Package format — ExperimentPackage v1 ({experimentId}.hdtpkg.zip)¶
package-descriptor.json # machine-readable descriptor (Jackson JSON)
manifest.yaml # copy of the human-readable manifest (documentation/provenance)
specification.md
research.xml # optional, if it exists
docker.env.example
code/{experimentId}.jar # compiled experiment classes (no fat-jar; runs on top of
# the classpath already present in admin-api: sdk, common, client, aggregator...)
SHA256SUMS # sha256 hash of each file above, in a deterministic order (SHA256SUMS last)
package-descriptor.json:
{
"packageFormatVersion": 1,
"experimentId": "exp-ecg5000-anomaly-001",
"contractVersion": "1.0.0",
"sessionFactoryClass": "br.ufma.lsdi.research.ecg.EcgSessionFactory",
"codeJarEntry": "code/exp-ecg5000-anomaly-001.jar"
}
manifest.yaml/specification.md/research.xml remain documentation/provenance (their hash is
recorded in SHA256SUMS, they are not parsed by code) — they do not duplicate the source of truth.
ExperimentPackageDescriptor¶
Record serialized/deserialized via Jackson ObjectMapper (toJson / fromJson, just like ClientPackageDescriptor). Note that PackageZipSupport.verifyChecksums tolerates packages without SHA256SUMS for legacy compatibility, whereas ClientPackageReader enforces SHA256SUMS presence. ExperimentPackageDescriptor rejects:
- a missing
packageFormatVersionor one other than1; - a blank or missing
experimentId,sessionFactoryClass, orcodeJarEntry.
All rejections throw FederatedException(ErrorCode.EXPERIMENT_PACKAGE_INVALID, ...).
ExperimentPackageWriter¶
Assembles the zip from a staging directory that already contains package-descriptor.json plus the
remaining files. Rules:
- Entries in deterministic alphabetical order;
SHA256SUMSis always the last entry. SHA256SUMSusesHashing.sha256(Path)(streaming, without loading large files into memory) — line format:<hash> <relative path>.- Fails if
code/{id}.jardoes not contain the.classentry corresponding to the declaredsessionFactoryClassin the descriptor (zip index check viaJarFile.getEntry(...), does not load the class) — catches the most common error before deployment.
ExperimentPackageReader¶
Extracts to a destination directory with the same protections as
LocalArtifactExperimentService.resolveSecure():
- Zip slip: each entry is resolved and normalized against the destination directory; entries that
escape (
../) are rejected before any write happens. - Zip bomb: each entry is copied with a configurable limit on decompressed bytes
(
ExperimentPackageReader(long maxEntryUncompressedBytes), default 200 MB) — the limit is enforced during the copy (actual bytes read), not trustingZipEntrymetadata. - Integrity: after extraction, it recalculates the hash of every file listed in
SHA256SUMSand compares them — a mismatch throwsFederatedException(ErrorCode.PACKAGE_HASH_MISMATCH, ...). - Format version: an unknown
packageFormatVersionis rejected byExperimentPackageDescriptor.fromJson(...)itself when parsing the extracted descriptor.
scripts/package-experiment.sh {experimentId}¶
sh scripts/package-experiment.sh exp-ecg5000-anomaly-001
# → target/packages/exp-ecg5000-anomaly-001.hdtpkg.zip
Optional flow with direct submission to the platform:
sh scripts/package-experiment.sh exp-ecg5000-anomaly-001 --submit
sh scripts/package-experiment.sh exp-ecg5000-anomaly-001 --submit --client-ids 1,2,3
sh scripts/package-experiment.sh exp-ecg5000-anomaly-001 --submit --platform-url http://127.0.0.1:8088
When --submit is used, the script sends multipart/form-data to POST /experiments/deploy
with the package and clientIds fields, reusing the same endpoint as the web monitor. The raw
HTTP response is saved to target/packages/{experimentId}.deploy.response.
Flow: mvn -pl research/experiments/{id} -am package -DskipTests (compiles the experiment's standalone
Maven module — prerequisite: its own pom.xml, artifactId equal to the experimentId, see
docs/components/13-research-experiments.md) → builds a staging directory by copying
manifest.yaml/specification.md/research.xml/docker.env.example from the experiment and the
newly compiled jar → writes package-descriptor.json → invokes ExperimentPackagerCli (a simple
main class, java -cp ... ExperimentPackagerCli <staging> <output-zip>, without exec-maven-plugin).
The experimentId → sessionFactoryClass/contractVersion mapping is a case table inside the
script itself — today it covers ECG5000, MIMIC v3.1, MNIST, and ergonomic-risk experiment packages. HAR/DT requires adding an
entry here and its own pom.xml for the experiment, following the same pattern.
Dynamic deploy — ExperimentDeploymentService (ADR-005)¶
ExperimentDeploymentService service = new ExperimentDeploymentService(executor, monitoringService);
ExperimentContract contract = service.deploy(packageZipBytes, List.of("1", "2", "3"));
Flow of deploy(byte[] packageZipBytes, List<String> clientIds):
- Writes the bytes to a temporary file and extracts them via
ExperimentPackageReaderinto a temporary staging directory (zip slip/bomb/hash/version checks already covered — see above). - Computes a
shortHash(12 characters of the SHA-256 of the zip's bytes) and moves the extracted content toExperimentRuntimePaths.deployedExperimentDirectory(experimentId, contractVersion, shortHash)— a directory versioned per deploy, preventing a redeploy from overwriting a jar in use by an isolated process already spawned for a running session. A redeploy of identical content (same hash) reuses the existing directory (idempotent). - Validates
sessionFactoryClassby launching a disposable isolated OS process (see "Process isolation (BL-236)" below) and sending it aVALIDATEcommand — the class is loaded and instantiated inside that separate process, never inside admin-api's own JVM, and the check confirmsinstanceof ExperimentSessionFactorybefore the process exits. - Registers an
IsolatedExperimentSessionFactory— never the loaded factory itself — that knows how to launch a fresh isolated process on demand for every subsequentcreate(...). AdminExperimentExecutor.registerFactory(experimentId, contractVersion, factory)— dynamic registration takes priority over the hardcoded legacy switch (HAR/ECG/DT continue working unchanged; the dynamic registry is consulted first increateSession/execute). Since BL-077, the registry and dynamic sessions are keyed by(experimentId, contractVersion), allowing two versions of the same experiment to coexist in the same process.- Creates an
ExperimentContractviaExperimentContractFactory.buildDraft(...).withStatus(ContractStatus.ACTIVE)— it is born ACTIVE (not DRAFT) because the goal is to deploy and train immediately; the metadata fields (coordinator, objective, digital twin type, task type) use generic values sincepackage-descriptor.jsondoes not carry those fields. AdminMonitoringService.registerExperiment(contract, clientIds)— the same path used byAdminApplicationBootstrap.registerDefaultExperiments().
Operational versioning¶
- The directory extracted on disk remains isolated by
contractVersion + shortHash. - The monitor now keeps a separate stateful session for each
(experimentId, contractVersion). - A redeploy of version
2.0.0does not automatically remove the already-running session of version1.0.0. - Session reset (
POST /experiments/{id}/reset-session?version=...) affects only the specified version. - A call with no explicit version in the executor (
execute(experimentId)) uses the most recently registered dynamic version for thatexperimentId, preserving compatibility with legacy flows.
Process isolation (BL-236)¶
Before this delivery, sessionFactoryClass was loaded and executed inside admin-api's own JVM
via URLClassLoader — the risk was documented and explicitly accepted (see "Accepted risk"
below, previous revision). This delivery replaces that with genuine OS-level process isolation:
br.ufma.lsdi.admin.isolated.IsolatedExperimentRunnerApp // entry point of the isolated process
br.ufma.lsdi.admin.isolated.IsolatedExperimentRunner // dispatch logic (testable without a process)
br.ufma.lsdi.admin.isolated.SubprocessIsolatedProcessLauncher // production launcher (real java -cp ... subprocess)
br.ufma.lsdi.admin.isolated.IsolatedExperimentSession // ExperimentSession proxy over the child process
br.ufma.lsdi.admin.isolated.IsolatedExperimentSessionFactory // ExperimentSessionFactory registered instead of the raw factory
- Protocol: one JSON line per command on the child's
stdin, one JSON line per response on itsstdout(IsolatedExperimentRequest/IsolatedExperimentResponse, encoded via the sharedbr.ufma.lsdi.common.util.JsonCodec). Commands:VALIDATE(load + instantiate only, no side effect — used byExperimentDeploymentService.loadFactory),LOAD(also callscreate(...), the resulting session is retained inside the child process),EXECUTE_ROUND,TRIGGER_EXTERNAL. stderris inherited from the parent (ProcessBuilder.Redirect.INHERIT) — library logging never shares the pipe used by the protocol onstdout, and never risks filling an undrained buffer and deadlocking the child.- Classpath: the child process is launched with the same
javabinary (ProcessHandle.current().info().command()) and the samejava.class.pathas admin-api itself — the deployed package is not a fat-jar, it expectsfederated-common/-client/-aggregator/etc. to already be on the classpath, exactly as the oldURLClassLoaderusedAdminExperimentExecutor.class.getClassLoader()as its parent. - Transparency to the rest of the system:
AdminExperimentExecutorneeded zero changes — it already depended only on theExperimentSessionFactory/ExperimentSessioninterfaces. Swapping "factory loaded in-process" for "factory that knows how to launch an isolated process" is entirely invisible to its 15+ existing tests. - Verified for real, not just at the interface level:
IsolatedExperimentRunnerAppIntegrationTest(@Tag("integration")) spawns an actualjavachild process (own PID, own JVM) withSubprocessIsolatedProcessLauncher, loads a fixture jar in it, executes a round, and confirmsdestroy()actually terminates the OS process (ProcessHandle.current().children()before/after) — no Docker dependency, ~1s total. - What is still out of scope: the HTTP deploy endpoint itself still has no authentication of its own (same trust model as every other admin-api route today — see "Accepted risk"); resource limits (CPU/memory/time) on the isolated process; and container-level isolation (Docker per deploy) as a stronger future option if the trust model around port 8088 ever changes.
MultipartFormParser¶
A minimal multipart/form-data (RFC 7578) parser with no external dependencies — used only by the
deploy endpoint, since the body contains a binary file (the existing AdminWebServer.form(...)
assumes application/x-www-form-urlencoded and would corrupt the upload). Returns text fields
(Map<String,String>) and file parts (List<FilePart> with field name, file name,
content-type, and bytes).
HTTP endpoint¶
| Method | Route | Description |
|---|---|---|
GET |
/experiments/deploy |
Upload form (package: file; clientIds: optional, CSV) |
POST |
/experiments/deploy |
Receives the multipart, delegates to ExperimentDeploymentService, redirects (303) to /experiments/{id}?version={contractVersion} |
The route is intercepted in AdminWebServer.handle(...) before the generic
path.startsWith("/experiments/") dispatch, because that generic path (routeExperiment) already
consumes the request body as a URL-encoded form — consuming the body twice, or as the
wrong type, would corrupt the binary upload.
Accepted risk (explicit user/operator decision)¶
| Risk | Status |
|---|---|
| RCE via upload — anyone with access to port 8088 can upload an arbitrary jar and have it loaded and executed, with no token, no origin restriction, no mTLS | Mitigated (BL-236). Uploaded code now always runs in a genuinely separate OS process (own PID, own JVM — see "Process isolation (BL-236)" above), never inside admin-api's own JVM. The endpoint itself still has no authentication — the monitor has none today for any other route either, and the operator chose to keep that trust model for deploy. Do not expose port 8088 outside a trusted network. |
| Zip slip during extraction | Mitigated — ExperimentPackageReader normalizes and validates each entry against the destination directory before writing. |
| Zip bomb | Mitigated — configurable limit on decompressed bytes per entry, enforced during the actual copy. |
| Corrupted/tampered jar in transit | Mitigated — SHA256SUMS verified after extraction, before the isolated process loads any class. |
| Version collision under an active session | Mitigated — extraction directory versioned by contractVersion + shortHash; an identical redeploy is idempotent, a redeploy with new content uses a new directory. |
| Regression in the hardcoded legacy HAR/ECG/DT path | Mitigated — the legacy switch in AdminExperimentExecutor was not removed; the dynamic registry is only consulted first, never replacing the fallback. |
| Isolated process left unbounded (CPU/memory/wall-clock) | Not yet mitigated — the child process inherits no resource limits from the launcher; a malicious/buggy package can still exhaust host resources. Documented as a follow-up, not blocking this delivery (it removes RCE against the monitor's own process/data, which was the accepted risk here). |
Explicit non-goal (unchanged): JVM sandboxing. SecurityManager has been deprecated since
JEP 411 (Java 17+) and is not a viable isolation option. Process isolation (this delivery) already
gives the meaningful guarantee — the deployed code can no longer read/corrupt admin-api's own
memory, credentials, or other experiments' in-process state; it runs as an unprivileged sibling
process instead. Container-level isolation (e.g., a Docker sandbox per deploy, with real resource
limits) remains a possible stronger future step if the trust model around port 8088 ever changes.
Manual verification¶
sh scripts/package-experiment.sh exp-ecg5000-anomaly-001
sh scripts/run-admin-web.sh
# Go to http://127.0.0.1:8088/experiments/deploy, submit
# target/packages/exp-ecg5000-anomaly-001.hdtpkg.zip, confirm that the experiment appears
# on the dashboard and that a round runs successfully.
Tests¶
| Module | Coverage |
|---|---|
federated-experiment-packager |
Experiment descriptor/writer/reader and simulation package tests; ClientPackageDescriptorTest, ClientPackageSignatureTest, ClientPackageWriterReaderTest, ClientPackagePackagerCliTest (client descriptor validation, Ed25519 signing, DEX/hash checks, safe archive reading/writing and CLI) |
federated-rest-api |
MultipartFormParserTest, AdminExperimentExecutorTest (dynamic registration, override, legacy fallback), ExperimentDeploymentServiceTest (fixture packaged at runtime via JarOutputStream, deploy validated through a fake isolated channel), ExperimentDeployHttpTest (end-to-end HTTP: real AdminWebServer + real multipart), br.ufma.lsdi.admin.isolated.* (BL-236: IsolatedExperimentRunnerTest, IsolatedExperimentSessionTest, IsolatedExperimentSessionFactoryTest — fake channel, no process; IsolatedExperimentRunnerAppIntegrationTest, @Tag("integration") — real java subprocess spawned end-to-end) |
research/experiments/exp-ecg5000-anomaly-001 |
EcgSessionFactoryTest (tag e2e: EcgSessionFactory.create(...) produces an executable session against a real MQTT broker and the ECG5000 dataset) |
Mandatory rules¶
- New packageable experiment: create its own
pom.xmlinresearch/experiments/{id}/(artifactIdequal toexperimentId) and a{Id}SessionFactory implements ExperimentSessionFactoryin the same package as the existing session. - Never remove the hardcoded legacy path in
AdminExperimentExecutorwhen adding a dynamic registration for an already-supported experiment. ExperimentPackageWriter/Readerare the only valid way to generate/read a.hdtpkg.zip— do not assemble zips manually for deploy.- Any new metadata field a dynamic
ExperimentContractneeds (coordinator, real objective, digital twin type) must be added topackage-descriptor.jsonand toExperimentPackageDescriptor, never hardcoded inExperimentDeploymentService.
Independent Android client algorithm package (.hdtclient.zip)¶
The signed client package format and its CLI are part of this same
federated-experiment-packager module. The Android client and REST deployment service consume the
descriptor/signature classes from this module; they do not bring a second packaging library into
the system.
The experiment/server package (.hdtpkg.zip) does not contain Android training code. The optional
Android algorithm package is a separate artifact, versioned by (experimentId, contractVersion),
and can be deployed without rebuilding or reinstalling the APK:
CLIENT_PACKAGE_SIGNING_KEY=/secure/release-key.pem \
CLIENT_PACKAGE_SIGNING_KEY_ID=release-1 \
sh scripts/package-federated-client.sh exp-mdpl-001 1.0.0 \
federated_mlp_classifier 1 br.ufma.lsdi.client.plugin.MlpTrainingPlugin \
path/to/classes.dex --submit
The command writes target/client-packages/{experimentId}-{contractVersion}.hdtclient.zip.
--submit sends it to POST /api/client-packages/deploy; the monitor verifies the Ed25519
signature and trusted key ID before making the package available through
GET /api/experiments/{experimentId}/client-package?version={contractVersion}. A different
package cannot replace an already deployed package for the same contract version; publish a new
contract version to update its algorithm.
The archive contains code/client-engine.dex (enforced code/*.dex entry by ClientPackageDescriptor and staged as code/client-engine.dex by ClientPackagePackagerCli), client-package-descriptor.json, client-package-signature.sig (Ed25519 signature), and SHA256SUMS.
The private key remains in the packaging environment. The server is configured with
HDT_AF_CLIENT_PACKAGE_TRUSTED_PUBLIC_KEY and HDT_AF_CLIENT_PACKAGE_TRUSTED_KEY_ID; the APK
pins the matching public key using CLIENT_PACKAGE_TRUSTED_PUBLIC_KEY and
CLIENT_PACKAGE_TRUSTED_KEY_ID.
On device, the package manager verifies the archive before using DexClassLoader. The plugin runs
inside the preinstalled MDPL OnArrivalPhenotypeInferenceEngine; local request/result transport
uses CDDL and never forwards training data to the remote MQTT broker. When no client package is
published, Android uses its built-in LocalTrainer. This packaging/deployment path is independent
of the .hdtpkg.zip experiment deployment described above; see
06-federated-client.md for the Android
runtime and the API routes.