Deploy and call an inference endpoint

Inference serves a trained LoRA through a project-owned endpoint. You can deploy an existing checkpoint without starting another training run. The endpoint has its own readiness, scaling and billing lifecycle.

Job-scoped API

Inference deployments are type=inference Jobs. One Trial records one worker process lifetime; requests reuse a worker, while an actual process restart creates a new Trial. Zero workers means zero running Trials. The OpenAI-compatible base URL is https://api.envloop.ai/v1/job/YOUR_JOB_ID/Inference (capital I).

Method	Path
GET	/v1/job/{job_id}/Inference/models
POST	/v1/job/{job_id}/Inference/chat/completions
POST	/v1/job/{job_id}/Inference/completions

The path selects the service. Use a model returned by this Job; models from another Job are rejected. The platform selects a worker. Do not pass trial_id, worker_id or upstream routing fields. The authenticated /inference/v1/* prefix returns 410 INFERENCE_ROUTE_RETIRED; POST requests are never redirected. Update the platform and client together. Endpoint management SDK/API methods remain available and return job_id; endpoints.generate() requires this explicit Job ID. The top-level el inference command has been removed. Local clients preserve /api: http://127.0.0.1:5173/api/v1/job/YOUR_JOB_ID/Inference.

Before you start

You need an authorized project Key, a registered training checkpoint, and a deployable serving profile matching its base model and revision. Install the public envloop-cli package from Installation for the CLI and Python SDK. A profile catalog entry is not a guarantee that every checkpoint is compatible; registration checks the actual artifact.

el auth status --check
el checkpoint list --limit 25
el job serving-profile list --json

A serving profile describes the base model/revision, supported adapter formats and rank, available hardware, default hardware and prices when provided. Use returned values. Do not guess a hardware name, assume a missing price is free, or infer deployment support from a model name alone.

Inspect in the Console

Jobs → Type: Inference lists services. Job Overview shows deployment details and Connect; Trials shows worker lifetimes. Adapters is a separate read-only directory under Jobs. Use the CLI commands below to register adapters, create endpoints and manage their lifecycle. The Console provides no deployment or lifecycle controls. Copy Base URL, Model and the connection example from Connect.

Deploy with Python

This example assumes a verified checkpoint already exists. It starts serving compute and retains the endpoint until you pause or delete it:

from envloop import Client

with Client.remote() as client:
    profiles = client.endpoints.profiles()["items"]
    profile = next(p for p in profiles
                   if p["name"] == "qwen38-27b@1" and p["deployable"])
    adapter = client.adapters.register(
        "YOUR_CHECKPOINT_PATH", serving_profile=profile["name"])
    endpoint = client.endpoints.create(
        name="support-chat", serving_profile=profile["name"],
        adapters=[{"alias": "sft", "adapter_id": adapter["adapter_id"]}],
        scaling={"min_replicas": 0, "max_replicas": 1,
                 "scale_to_zero_after_s": 300})
    print({"job_id": endpoint["job_id"], "endpoint_id": endpoint["endpoint_id"],
           "model": endpoint["model"] + "/sft"})
    client.endpoints.wake(endpoint["endpoint_id"])
    ready = client.endpoints.wait(endpoint["endpoint_id"], timeout=2400)
    print(ready["status"])

Use a fresh endpoint name or inspect the existing endpoint before retrying creation. wait() only observes readiness; it does not wake a scaled-to-zero endpoint. For experiments with cleanup on failure, use Train and deploy.

The CLI accepts the same public registration/create request as a JSON file:

el job adapter register --file adapter.json
el job submit --type inference --file endpoint.json
el job get YOUR_JOB_ID
el job wake YOUR_JOB_ID
el job event YOUR_JOB_ID --json

adapter.json contains request_id, serving_profile and source: {"type":"train_checkpoint","tinker_path":"YOUR_CHECKPOINT_PATH"}. endpoint.json contains request_id and spec; put name, serving_profile, a hardware value from that profile, adapters and scaling inside spec. Use a new request ID for a new operation and reuse it only for an identical retry.

Call from Python

Install the OpenAI Python client in your application's environment and set ENVLOOP_API_KEY through your normal secret handling. Copy the exact model from your endpoint detail:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.envloop.ai/v1/job/YOUR_JOB_ID/Inference",
    api_key=os.environ["ENVLOOP_API_KEY"],
    timeout=120,
)
response = client.chat.completions.create(
    model="YOUR_ENDPOINT_MODEL/sft",
    messages=[{"role": "user", "content": "Explain gravity in one sentence."}],
    max_tokens=128,
    temperature=0,
)
print(response.choices[0].message.content)

YOUR_ENDPOINT_MODEL/sft is a placeholder. The actual model is the returned endpoint model prefix plus your alias; do not use endpoint_id, adapter_id, the base-model repository name or checkpoint URI here. Custom deployments should use the base URL shown by their console.

The API provides OpenAI-compatible chat completions, text completions and model discovery. Streaming requests are supported, but model-specific features such as tools, image input, structured output or extra generation parameters depend on the selected serving profile and model. Compatibility does not include every OpenAI product API.

For streaming text:

stream = client.chat.completions.create(
    model="YOUR_ENDPOINT_MODEL/sft",
    messages=[{"role": "user", "content": "Say hello."}],
    max_tokens=64,
    stream=True,
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)

Make an HTTP request

Set ENVLOOP_MODEL to the complete model alias shown by your endpoint. The request reads your existing environment Key; do not paste a real Key into documentation or a shared script.

curl --fail-with-body https://api.envloop.ai/v1/job/YOUR_JOB_ID/Inference/chat/completions \
  -H "Authorization: Bearer $ENVLOOP_API_KEY" \
  -H 'Content-Type: application/json' \
  --data "{\"model\":\"$ENVLOOP_MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hello.\"}],\"max_tokens\":64}"

A successful HTTP response with non-empty assistant content proves the requested endpoint answered. To check trained behavior, compare the /sft and /base aliases on held-out prompts using the same generation settings.

Anthropic Messages API

The same endpoint also accepts Anthropic Messages requests at POST https://api.envloop.ai/v1/job/YOUR_JOB_ID/Inference/v1/messages, so Anthropic SDKs and Claude Code can use your trained model. Set the base URL to https://api.envloop.ai/v1/job/YOUR_JOB_ID/Inference (the client appends /v1/messages) and authenticate with your EnvLoop key as X-API-Key or Authorization: Bearer, not both.

import os
from anthropic import Anthropic

client = Anthropic(base_url="https://api.envloop.ai/v1/job/YOUR_JOB_ID/Inference",
                   api_key=os.environ["ENVLOOP_API_KEY"])
message = client.messages.create(model="YOUR_ENDPOINT_MODEL/sft", max_tokens=128,
                                 messages=[{"role": "user", "content": "Explain gravity in one sentence."}])
print(message.content[0].text)

Streaming and tool use follow the serving profile's capabilities. Claude Code works with the endpoint when ANTHROPIC_BASE_URL, ANTHROPIC_API_KEY and ANTHROPIC_MODEL (and the ANTHROPIC_DEFAULT_*_MODEL aliases) point at it and CLAUDE_CODE_EFFORT_LEVEL is a level the model accepts, such as medium. Token counting (/v1/messages/count_tokens) is not provided. To run Claude Code on the model, see Run an Agent benchmark on a trained model.

Understand cold starts

State	What to do
SCALED_TO_ZERO	No running replicas; a request or Start action can allocate one
PROVISIONING	Wait for model preparation; a request can return a cold-start response
READY	Send a request; inspect ready replicas if failures occur
PAUSED	Resume explicitly before requests can start compute
SUSPENDED	Resolve the displayed error or billing condition before resuming
FAILED	Inspect error and activity; a replacement may be required
DELETING	Deletion and release are still in progress

A cold endpoint can return HTTP 503 with ENDPOINT_COLD_STARTING and Retry-After. Wait at least that interval and retry with a bounded overall deadline. Do not treat repeated immediate requests as a faster way to start the model. endpoints.generate() exposes cold-start failures rather than hiding them in an unbounded retry loop.

Pause, resume and delete

el job pause YOUR_JOB_ID
el job get YOUR_JOB_ID
el job resume YOUR_JOB_ID

Pause blocks new requests from starting replicas; running replicas are released asynchronously. With minimum zero, resume allows requests again but can leave the endpoint at zero until it is used. To warm it explicitly, use wake and observe readiness.

Idle scale-to-zero allows future requests to restart compute. Pause is appropriate when you want that automatic restart blocked. GPU charges apply while replicas run, including startup, and stop after release. The displayed rate is not the final bill.

When permanently finished, delete in dependency order:

el job stop YOUR_JOB_ID
el job adapter delete YOUR_ADAPTER_ID --wait
el checkpoint delete YOUR_CHECKPOINT_RESOURCE_ID --wait

Delete only the resources you intend to remove. Job stop is permanent: it reports cancelling until compute release is confirmed and then cancelled. Read inference.cleanup_operation_id with el setting operation get. Stopping retains its adapter and checkpoint; shared/dependent resources can block later deletion. Cleanup returns a persisted Operation, so inspect completion and errors rather than assuming a successful request already freed resources. Keep the checkpoint if you plan to redeploy or resume training.

Diagnose a failed call

Error or symptom	Check
401	EnvLoop Key and correct API origin
403	Project membership, Key scope and inference permissions
Model not found	Full model prefix/alias, correct project and whether the endpoint still exists
Adapter validation failed	Checkpoint availability, model revision, rank, format and validation details
503 cold start	Retry-After, provisioning state and bounded retries
Paused/suspended	Explicit resume and the displayed reason
Empty or cut-off answer	max_tokens, model template and returned finish reason
Unexpected base behavior	Verify the selected alias is your trained adapter, not /base

Next: Checkpoint management or SDK reference.