Skip to main content

GPU Runner training

Before you start​

Update the client and check the submission options in your active environment:

python -m pip install --upgrade envloop-cli
el job submit --help

Before using the shortcuts below, confirm that help lists --trials, --gpu-type and --gpu. If it does not, check that el comes from the Python environment you just updated, or use the request-file or SDK workflow on this page.

Your project needs training.run permission, access to a published GPU Runner Revision and available GPU capacity. CLI, request-file and SDK submissions all require platform support for Runner training. If submission is refused, use the error code to check permissions, the Runner or capacity, and contact your project administrator when needed.

In Console, open Runners, then select a name or its Versions count to browse that Runner's published revisions. Select a Revision to view its execution definition and files; use Versions on the detail page to switch revisions. Versions retain their full immutable Git commit identifiers. Console browsing is read-only; submit Jobs through the CLI below.

Use el runner list --json to find a Runner you can read. Pin its full namespace/name@revision selector. The platform checks project training.run and Runner access before freezing the published files. It always executes that Revision's main.sh or main.py entrypoint.

Use a request file or SDK​

Confirm that el job submit --help lists --spec. If it is missing, update the client in the Python environment that provides el before continuing.

--spec reads a Job specification JSON file and accepts explicit Runner/resource options as higher-priority overrides for every Trial (unspecified resource keys are preserved); --trials remains mutually exclusive. Save a request such as this as gpu-run.json; substitute a full published commit:

{
  "request_id": "gpu-run-001",
  "trials": [
    {"name": "first", "spec": {"runner": "envloop/bash@FULL_COMMIT", "resources": {"gpu_type": "H100", "gpus": 4}, "timeout_seconds": 600}},
    {"name": "second", "spec": {"runner": "envloop/bash@FULL_COMMIT", "resources": {"gpu_type": "H100", "gpus": 4}, "timeout_seconds": 600}}
  ]
}
el job submit --type train --spec gpu-run.json

The SDK accepts the same request through client.training_jobs.submit(request). A single Trial can use {"request_id":"gpu-single-001","spec":{"runner":"envloop/bash@FULL_COMMIT"}}. The HTTP request adds "type":"train" at POST /v1/jobs. Recipe and Runner specifications cannot mix; Runner mode rejects Dataset/Task inputs, uploaded files and client-supplied resolved references.

Keep the same request ID and identical input when retrying an uncertain submission. The first authorized Runner snapshot is retained even if its source ref later moves. Changing Trial count, resources or any explicit input conflicts; use a new ID. Revoked Runner access remains enforced.

Copy the returned job_id for the reads below. Choose one submission method per run to avoid starting duplicate Jobs. Each Trial runs independently; reduce the Trial count or GPU count for a smaller first run when the program supports it.

Optional CLI shortcuts​

If el job submit --help lists the shortcut options, you can submit the same work without a request file:

el job submit --type train \
  --runner envloop/bash@FULL_COMMIT \
  --trials 2 --gpu-type H100 --gpu 4 \
  --docker YOUR_PUBLIC_GPU_IMAGE \
  --cpu 8 --memory 64Gi --disk 100Gi --shm-size 16Gi \
  --command 'python -c "import torch; print(torch.cuda.device_count())"' \
  --timeout-seconds 600 --request-id gpu-run-001

The spellings are exactly --trials, --gpu-type and --gpu. This submits one Job with two independent Trials, each requesting four H100 GPUs on one node. If both run together, they need eight GPUs. Queueing, quotas and budget determine actual concurrency; these Trials do not form a distributed cluster. The default Trial count is one; the maximum is 16. Counts must be positive integers.

--docker selects the image. Omit it to use the Runner's image; the platform resolves and freezes an immutable digest before allocation. Images must be publicly pullable Linux GPU images with Bash and supported Python and must satisfy the platform bootstrap and driver requirements. Application dependencies and extra files belong in your prebuilt image. This workflow does not scan, package or upload a local directory, script or source file.

--command is an optional run_command parameter declared by the published Runner. It does not replace the entrypoint. Unsupported Runners reject it. Commands preserve multiline text and quotes, must contain nonwhitespace text, must have no NUL, and are limited to 65536 UTF-8 bytes. Omission runs the Revision's default logic. No Profile, custom parameter map or environment override is accepted in this initial Runner mode.

GPU type and count, CPU, memory, disk, shared memory and timeout apply to each Trial. Omitted values come from the Runner and server policy. An effective GPU request is required; supported hardware, quantities, budget and limits are checked by the server. Clients do not select an executor, provider or backend.

Verify execution and release​

el job get JOB_ID
el trial list --type train --job-id JOB_ID --json
el job log JOB_ID --follow
el job result JOB_ID
el job stop JOB_ID

Inspect each Trial's frozen Runner reference, image digest, effective resources, published entrypoint, command, state, raw exit code, output status and independent cleanup_confirmed. Nonzero exit remains failure. Exit zero with failed output collection does not count as complete success. Logs preserve real stdout/stderr and EOF; arbitrary output does not become fabricated loss or metrics. Files declared in execution.outputs are stored as artifacts with upload status, size and SHA-256; an artifact is not automatically a resumable Checkpoint.

A failed Trial does not cancel siblings. Stopping a Job cancels all its queued/running Trials, collects bounded output, and releases each allocation. Once user code starts, the platform does not transparently replay it after execution loss. To run it again, submit a new Job.

Console Experiment provides read-only views. Submit and manage Jobs through the CLI or SDK, then check each Trial’s output and cleanup state.

Tags​

The request-file and SDK forms accept optional tags inside each Trial spec. Use a string map with at most 16 entries; keys match [A-Za-z0-9_][A-Za-z0-9_.-]{0,63}, and values contain at most 256 printable characters. For example, add "tags": {"experiment": "gpu-baseline", "purpose": "sft"} beside runner. Submit using el job submit --type train --spec REQUEST.json; there is no standalone --tags flag. Job and Trial responses expose tags, shown in the Console tag panel. For multiple Trials, the Job summary shows the first Trial’s tags; inspect training.trials for each Trial’s tags. This addition requires the matching platform implementation.

Fixed Job Profile​

A compatible Runner can receive an envloop.job-profile/v1 snapshot through spec.profile: inputs contains authorized content references pinned by manifest digest, recipe holds fixed parameters, and plugins binds approved packages. The platform validates content and authority before allocating GPUs. The client never uploads local paths or selects a provider. Represent decimal recipe parameters as strings.

Profile v2 separates an executable Recipe reference from its parameters. Use the published full manifest SHA-256 for each reference, and select a Runner revision with the matching envloop.train-recipe/1 ABI and Recipe entrypoint. For example, set spec.profile to the following shape, replacing the digest placeholders with the publication's actual digests:

{
  "schema": "envloop.job-profile/v2",
  "recipe": {
    "uri": "envloop://recipes/envloop/pev-recipe@<recipe-manifest-sha256>",
    "manifest_sha256": "<recipe-manifest-sha256>"
  },
  "parameters": {"steps": 1},
  "inputs": {
    "data": {
      "uri": "envloop://datasets/envloop/pev-smoke-data@smoke-1",
      "manifest_sha256": "<dataset-manifest-sha256>",
      "config": "default",
      "split": "train"
    }
  },
  "plugins": []
}

The Recipe fixes its source dependencies and parameter schema. The platform checks every dependency's permissions and freezes validated defaults together with your parameters. Recipe source belongs to recipes; Dataset inputs belong to datasets; outputs remain artifacts. This requires the matching platform and published Runner implementation.

Declared outputs are streamed and read back for verification. Compatible LoRA output containing adapter_config.json and adapter_model.safetensors can be registered as a durable Checkpoint for inference; ordinary artifacts do not automatically become models. Inspect the Job result's Checkpoints and independent cleanup_confirmed.