Error codes

Every refusal from the public API is a JSON object with a stable code, a readable message and the request_id of the call. When the platform knows what you should change it also sends a hint and details: flat records that name the affected task and the values that matter, such as a size and its limit. el prints all of them.

{
  "code": "DATASET_TASK_ARCHIVE_TOO_LARGE",
  "message": "Task acme/claims-0042: The task archive is 40.0 MiB; a task archive can be at most 16 MiB.",
  "hint": "Move large files out of the task directory (into the Runner or the task image), or split the task, then publish a new Dataset version.",
  "details": [{"member_index": 3, "task": "acme/claims-0042", "size_bytes": 41943040, "limit_bytes": 16777216}],
  "request_id": "6f1c…"
}

Branch on code, not on message. Messages can be reworded; codes are not renamed. A code you do not recognize still comes with a message and an HTTP status; read the status table in HTTP API.

Job submission

A submission is checked task by task before the Job is created. A refused submission creates no Job: fix the input and submit again.

Code	HTTP	Meaning	What to do
DATASET_TASK_ARCHIVE_TOO_LARGE	422	A task's published archive is bigger than the per-task delivery limit of 16 MiB. details has size_bytes and limit_bytes.	Move large files out of the task directory: put tools and models in the Runner or the task's container image, and fetch large data at run time. Or split the task. Then publish a new Dataset version and submit it.
DATASET_DELIVERY_KEY_MISMATCH	422	The published archive record of a task does not belong to the selected Dataset version, config, split and task.	Republish the Dataset version. If it persists, report the request ID.
DATASET_DELIVERY_INVALID	422	A task's archive record is incomplete or malformed. details.reason says which part: delivery_missing, size_invalid, sha256_invalid, fields_unexpected, url_not_allowed, reference_invalid or name_invalid.	Republish the Dataset version with a current EnvHub client. name_invalid means a Dataset, version, config, split or task name uses characters other than letters, digits, ., _ and -.
INVALID_RUNNER_ENVIRONMENT	422	The Runner or Task [environment] table has an invalid value; the message names the field.	Fix the field and publish a new version.
RUNNER_UNAVAILABLE	422	The platform could not prepare a task for an unexpected reason. details names the task.	Retry once. If it repeats, report the request ID: the operator has the full diagnostic.
DATASET_CONFIG_REQUIRED	409	The Dataset has several configurations and none is the default.	Pass --config NAME.
COMPOSE_RUNNER_DISABLED	422	Compose (multi-container) Runners are not enabled on this platform.	Ask the platform operator to enable them for your project, or use a non-compose Runner.
COMPOSE_RUNNER_DENIED	422	This compose Runner revision, project or Task shape is not approved.	Ask the platform operator to approve it, or choose an approved Runner (el runner list).
COMPOSE_TASK_REQUIRED / COMPOSE_RUNNER_REQUIRED	422	A compose Runner needs multi-container Tasks, and multi-container Tasks need a compose Runner.	Match the Runner to the Dataset.
AGENT_PROFILE_DENIED	422	The Profile's Agent, model or budget parameters are not accepted; the message says which.	Fix the Profile (el profile get) and submit again.
MODEL_ACCESS_DENIED	422	The Runner has model access in this project, and the Job sets an explicit command. Granted access covers only the Runner's own entrypoint.	Submit without --command, or ask a project Admin to revoke the grant.
MODEL_ACCESS_PROVIDER_UNSUPPORTED, MODEL_ACCESS_UNAVAILABLE	422	The Runner has model access in this project, but the Job pins a provider that cannot deliver the key, or the grants cannot be read now. No Job was created.	Submit without pinning a provider, or retry later. See Model access for Runners.
UNSUPPORTED_CAPABILITY, DIND_UNAVAILABLE, TEMPLATE_NOT_READY	422	The selected execution provider cannot run this Runner or Task shape yet; the message says why.	Change resources, timeout or kind, retry later for TEMPLATE_NOT_READY, or report the request ID.

Train Job submissions are refused with these codes; no Job is created:

Code	HTTP	Meaning	What to do
INVALID_TRAIN_JOB_SPEC	422	The request does not match the recipe's schema: a field, value or record is invalid, two fields conflict, or dataset and records are both set. details name each field (with record_index or member_index where it applies) and the rule, never the values.	Fix the fields in details, following hint when present. Compare the request with request_schema from GET /v1/training-models.
TRAINING_DATA_UNAVAILABLE	422	The training data reference is not readable: the dataset is not a record Dataset version of this project, has no such split, was retired, or does not match its pinned manifest_sha256, sha256 or row_count.	Check el data list --type dataset and the split name, then submit again.
TRAINING_DATA_TOO_LARGE	422	The selected Dataset split is larger than one Train Job reads: 64 MiB or 10,000 rows.	Split the data across Jobs, or continue from a parent checkpoint.
TRAIN_JOB_IDEMPOTENCY_CONFLICT	409	The request_id already belongs to a Train Job with different inputs.	Resend the original request, or choose a new request_id.

The 16 MiB limit is fixed by the platform. Publishing a Dataset through the platform's EnvHub release (envhub:deploy) now refuses such a task before upload; versions published earlier can still contain one.

Older platform versions answered all three delivery refusals with INVALID_DATASET_DELIVERY, and some refusals with the generic INVALID_REQUEST.

Trial failures

A Trial that fails after it started records its own error code. el job get, el job result and el trial get print one line per failed Trial, for example:

Trial trial_01… failed: EXECUTOR_INTERNAL_ERROR during scheduling - PermissionError: [Errno 13] Permission denied

The same fields are in the JSON: error, error_message (one line, when the platform recorded one) and failed_phase (the phase the Trial was in: scheduling or running). Job events carry them too. A Train Job also shows its code in the Job's training.error_code.

Code	Meaning	What to do
EXECUTOR_INTERNAL_ERROR	The platform hit an unexpected error while preparing or running the Trial. error_message gives the error type. This is not caused by your task.	Retry the Job. If it repeats, report the Job ID and the time: the operator has the full diagnostic.
EXECUTOR_PERMISSION_DENIED, EXECUTOR_FILE_NOT_FOUND, EXECUTOR_IO_ERROR, EXECUTOR_TIMEOUT	The platform could not read or write one of its own files or directories, or a platform operation timed out. This is a platform problem, not your task.	Report the Job ID and the time; retrying usually fails the same way until the operator fixes it.
DATASET_DELIVERY_UNAVAILABLE	The platform could not authorize the task download.	Retry later; report it if it persists.
TIMEOUT	The Trial ran longer than its time limit.	Raise the Runner timeout or make the task faster.
SANDBOX_START_FAILED	The platform created the Trial's sandbox but it did not start: it stopped during boot, its image has no usable /bin/sh, or the provider kept it waiting for longer than the platform's start limit (for example no capacity, or a large image that pulled too slowly or could not be pulled). The sandbox is removed and nothing ran, so it is not billed. This is an infrastructure failure, not a result of your task.	Submit again with a new request_id. If it repeats for the same Runner, check that its image is public or otherwise pullable by the platform, then report the Job ID.
EXECUTION_QUEUE_TIMEOUT	The Trial or Train Job waited for an execution slot longer than the platform's queue limit, 30 minutes by default (platform setting EXECUTION_QUEUE_TIMEOUT_SEC). An Agent or Sandbox Trial waits while its execution provider or its project's concurrent Trial limit is full. Time spent waiting for its own Job's budget does not count. In a Train Job with several Trials, a later Trial waits for the Trial before it, and its limit counts from that Trial's end. Nothing ran and no GPU was allocated.	Submit again later with a new request_id.
TRAINING_TIMEOUT	A Train Job Trial reached the time limit of its recipe and the platform stopped it. The limit counts from when the Trial started, and each Trial of a Job gets the whole limit. Time in the queue does not count. The Job and its Trial end failed. A stop you request ends cancelled without this code.	Read the recipe's limits.timeout_seconds from GET /v1/training-models (Run-time limits). Reduce max_steps, records or max_length, then submit a new Job with a new request_id.
RUNNER_COMMAND_FAILED	The Runner command exited with an error.	Read the Trial logs with el job log JOB_ID.
MODEL_ENV_SETUP_FAILED	The sandbox started but could not receive the Runner's model credential file. This is an infrastructure failure, not a result of your task.	Submit again with a new request_id; report the Job ID if it repeats.
MODEL_ACCESS_UNAVAILABLE, MODEL_ACCESS_PROVIDER_UNSUPPORTED, LLM_KEY_UNAVAILABLE	The Runner has model access in this project, but the platform cannot deliver it for this Trial now. Nothing started.	Read model_access in the project's execution settings, then ask the platform operator. See Model access for Runners.
INSUFFICIENT_BALANCE	The workspace has insufficient available balance for the run. For an Inference Job or endpoint, details names the up-front replica reservation (Submit an Inference Job).	Ask a workspace member with billing permission to review the request before retrying.
BUDGET_EXCEEDED	The Job reached its budget.max_cost_usd. For a Train Job it also means a Trial did not start because the remaining budget could not cover its whole time limit. The Trial ends cancelled; a stopped Train Trial registers no Checkpoint.	Read the Job's cost. For a Train Job compare the cap with limits.max_trial_cost_usd from GET /v1/training-models, then submit a new Job with a new request_id and a larger cap or fewer Trials (Train Job budget).
BUDGET_UNMETERED	Some of the Job's usage, or the GPUs of a Train Job's recipe, have no price, so the budget cannot be enforced. Nothing new started.	Submit without a budget, or report the Job ID.
EXECUTION_FAILED	The failure has no public code.	Read the Trial logs; report the Job ID if the logs do not explain it.

Error messages never contain URLs, file paths, storage keys or credentials. When the platform cannot say more without revealing one of these, it says only the error type and keeps the details for the operator.

Train Job failures

A failed Train Job shows its code in training.error_code, and its Trial carries the same code in error. The Trial's failed_phase is scheduling when it failed before training started running, and running after. error_message is the fixed line for the code. The codes above (TRAINING_TIMEOUT, EXECUTION_QUEUE_TIMEOUT, INSUFFICIENT_BALANCE) apply too.

Code	Meaning	What to do
GPU_JOB_FAILED	The GPU job failed. During scheduling it never started training, for example because no GPU could be provisioned; during running the training process stopped with an error.	During scheduling, submit again later with a new request_id: this is a platform or capacity problem, not your data. During running, read el job log JOB_ID --source runtime for the training error, then fix the data or options. Report the Job ID if the logs do not explain it.
CHECKPOINT_NOT_DURABLE	Training finished, but its checkpoint could not be saved to durable storage.	Report the Job ID. Submit again with a new request_id if you need the checkpoint now.
BATCH_INPUTS_UNAVAILABLE	The platform could not prepare the training inputs, so no GPU was allocated. training.error_detail names the input, record and rule that failed.	Fix the named record or option and submit a new Job.
BATCH_DRY_RUN_NOT_EXECUTED	Only on deployments that plan Train Jobs without GPUs: the inputs were packaged and no GPU job was launched. The Job ends cancelled.	Nothing; use a deployment that runs training.
PRICE_MISSING	The GPU the recipe needs has no price on this platform, so it could not be reserved.	Report it to platform support.
RESERVATION_CONFLICT	The billing reservation for the Trial conflicted with an existing one.	Submit again with a new request_id; report the Job ID if it repeats.
TRAINING_BILLING_REQUIRED	Training billing is not available for the workspace, so the Trial did not start.	Contact platform support.
TRAINING_BACKEND_UNAVAILABLE	Runner training is not available on this platform, so the Trial did not start.	Contact platform support.
PROFILE_AUTHORIZATION_UNAVAILABLE	The Trial's Profile inputs are no longer authorized for the project, so it did not start.	Check the Profile (el profile get) and its inputs, then submit again.
TRAIN_RUNNER_LAUNCH_REJECTED	The platform refused to launch the Runner training container.	Check the Runner and its resources against Runner training, then submit again.
TRAIN_RUNNER_TIMEOUT	The Runner training container ran past its timeout and was stopped.	Raise timeout_seconds or shorten the run.
TRAIN_RUNNER_PROCESS_FAILED	The Runner training command exited with a nonzero code.	Read the Trial logs with el job log JOB_ID.
TRAIN_RUNNER_RESULT_INCOMPLETE	The Runner training container ended without a complete result.	Read the Trial logs; check that the command writes its declared outputs.
TRAIN_RUNNER_OUTPUTS_FAILED	The Runner's declared outputs could not be collected.	Check execution.outputs and the files the command writes.
TRAIN_RUNNER_CHECKPOINT_FAILED	The Runner's checkpoint output could not be registered.	Check the checkpoint output against Runner training.
NATIVE_TRAIN_FAILED, NATIVE_TRAIN_INTERRUPTED	Local deployments only: the in-process training failed, or a restart interrupted it.	Submit again with a new request_id.
TRIAL_CLEANUP_UNKNOWN	The platform could not confirm that the Trial's resources were released.	Report the Job ID; do not assume the resources are gone.

Waiting and unconfirmed states

These codes describe a Job that is not finished yet. They are not failures.

Code	Where	Meaning	What to do
EXECUTION_RECONCILING	A Trial's error while the Trial is not finished; training.error_code of a Train Job	The platform is confirming what happened to a Trial's execution with the provider, for example after a restart. The Job's status is unknown until it knows.	Do not submit the same work again. Read the Job later; it moves to its real state when the result is confirmed.
BATCH_PROVIDER_UNAVAILABLE	training.error_code of a Train Job that is not finished	The platform could not read the GPU job's state from the provider. The Job's status is unknown until a read succeeds.	Same as EXECUTION_RECONCILING.

unknown is not a final status. A Job shows it while any of its unfinished Trials has EXECUTION_RECONCILING, or while a Train Job has one of the two codes above. A Job that is cancelling, or already finished, keeps that status instead.

Service time-outs

The API returns these codes when the platform service behind it does not answer.

Code	HTTP	Meaning	What to do
UPSTREAM_TIMEOUT	504	The API forwarded the request, but the platform did not answer in time. The request may still have been processed.	Read the resource before you retry a write. For a Job submission, check el job list. If the submission had a request_id, you can also send it again unchanged: an identical retry returns the Job that already exists.
CANCELLATION_TIMEOUT	504	A stop request got no confirmation in time. The Job may still be stopping.	Read the Job status before you stop it again.
SOURCE_UNAVAILABLE	503	The API could not reach the platform service, or the service failed unexpectedly.	Retry later. Before you retry a write, read the resource as for UPSTREAM_TIMEOUT.

Older platform versions answered a time-out with 503 SOURCE_UNAVAILABLE.

Connection failures in el

These codes come from the client, not from the API.

Code	Meaning	What to do
NETWORK_ERROR	The client could not connect: DNS, TLS, proxy or the service is unreachable. Nothing was sent.	Check the API URL and your network, then retry.
REQUEST_TIMEOUT	The request was sent but no answer arrived in time.	The request may still have been processed. Read the Job or resource (el job list) before you submit again.
RESPONSE_INTERRUPTED	The connection closed before the answer was complete. The service may have failed or restarted.	Same as REQUEST_TIMEOUT. Report the time if it repeats.

When you report a problem, include the code, the HTTP status and the request_id. See Troubleshooting.