Evaluate and compare results

An evaluation needs both a reproducible input and an interpretable result. EnvLoop records execution status, verifier reward, structured output and score coverage separately so you can tell a failed run from a low-scoring model or a missing measurement.

Run a small evaluation first

1. Choose a readable Dataset revision and its pinned Runner.
2. Select a Profile containing the model/agent parameters that the Runner expects. A Profile field has an effect only if that Runner reads it.
3. Submit one Task first, inspect its logs, reward and artifacts, then submit the complete Dataset when the setup works.
4. Use the same fixed Dataset, Runner, task set and attempt count for comparisons. Record the Profile, model and request ID as well.

el job submit --dataset DATASET_NAME@VERSION --attempts 2 --profile team/eval@1
el job get JOB_ID
el job result JOB_ID

Replace all resource placeholders with catalog values. Attempts are independent execution slots and increase the amount of work. A successful submission starts execution; it does not prove the agent passed a benchmark.

Read the result in order

Open Jobs → your Job → Metrics. Start with the number of completed and scored attempts, then inspect aggregate metrics and per-Task results. Open an individual Trial when a reward is missing or unexpected.

Signal	Question it answers
Job/Trial status	Did execution finish, fail or get cancelled?
Exit code and error	What execution failure was reported?
Verifier reward	What score did the declared evaluation return?
Structured result	What additional dimensions or judge summaries were produced?
Score coverage	How much of the requested evaluation actually produced a numeric reward?
Artifact persistence	Are the files available for independent inspection/download?

completed and succeeded are execution outcomes. They do not imply reward 1. failed Jobs may still contain valid scored attempts. A returned checkpoint or artifact reference also does not by itself establish a successful evaluation.

Aggregate metrics

Field	Meaning
score	Arithmetic mean of attempts with a numeric reward
pass	Mean binary reward when every scored reward is 0 or 1
pass@k	Fraction of fully scored members with at least one passing attempt
pass^k	Fraction of fully scored members for which all requested attempts pass

Here k is the requested attempts per member. Only members with exactly that many dispatched attempts, all scored, enter the two k metric denominators. No fully scored members means both are null. Nonbinary rewards make the binary metrics null; no scored attempts makes all metrics null. Console shows the actual k, such as pass@2 / pass^2, and formats pass rates as percentages. Job scores, per-Task results and the project leaderboard share one primary-score display: values from 0 to 1 use percentages (0.75 → 75%, 1 → 100%); values outside that range keep their original scale. The API score remains the mean reward. Separate composite scores and dimensions retain their own units.

For example, with two attempts per Task, suppose A has rewards [1, 0], B has [1, 1], and C has [0, missing]. The API mean score and pass are 3/5 = 0.6, displayed as 60% in Console. Only A and B enter the k metrics, so pass@2 = 1.0 and pass^2 = 0.5. C's missing attempt is not silently converted to zero, and the high pass@2 does not mean all requested Tasks were fully evaluated.

Relation to benchmark estimators

With n binary evaluations of one Task and c successes, the general formulas are pass@k = 1 - C(n-c,k)/C(n,k) and pass^k = C(c,k)/C(n,k), for 1 ≤ k ≤ n. Average the per-Task estimates with equal Task weight. For n=5, c=3 and k=2, the results are 90% and 30%; substituting the measured success rate into 1-(1-p)^k or p^k gives a different estimate.

Jobs currently expose the case n=k: at least one success versus all attempts succeeding. They do not expose a curve for independently chosen k values. See the Harbor pass@k implementation used by Terminal-Bench and τ-bench pass^k implementation. Harbor counts absent verifier rewards as failures; EnvPlatform preserves them as unknown and reports coverage. Compare scores only with matching missing-result policies and evaluation protocols.

Always include coverage

The API's score_counts accompanies each aggregate:

• total: requested Tasks × attempts per Task, including undispatched slots.
• scored: attempts with a numeric reward.
• missing: requested attempts without a reward, including undispatched, waiting or running attempts.
• invalid: terminal attempts without a reward.
• total_members: selected Dataset members.
• complete_members: members with every requested attempt scored.

Report “score 0.6, 5/6 attempts scored, 2/3 members complete” rather than just “score 0.6”. Compare failure/missing counts alongside scores, especially when models have different timeout rates. Partial coverage cannot establish a result on the whole test set.

Structured metrics

A supported declared result artifact can provide a composite score, named dimensions and judge summaries. The Trial Metrics tab shows these alongside the reward. They do not replace the verifier reward used for Job ranking. The original output remains in Artifacts if the structured projection is unavailable or invalid.

When comparing custom dimensions, confirm that names, scale, scoring direction and verifier versions match. A field named accuracy is not enough to establish that two independent Runner implementations measure the same quantity.

Read the project leaderboard

Read saved results, ranks, scores and coverage in Jobs → Metrics → Project leaderboard. The Console provides no action to add results.

The board compares explicitly submitted Jobs with the same frozen Dataset revision, Runner reference, Task names/digests and attempt count. Model/Profile choices may vary. Rows include rank, model or Job ID, score, scored count, missing count and available composite metrics. Open a row to inspect its source Job.

A failed Job with partial scores can appear; read its missing count before treating its rank as meaningful. Adding the same Job again returns its saved snapshot. Results are visible within the project and are not automatically published to the public website. See Research and leaderboards for the separate public submission workflow.

Turn failures into useful next runs

For an execution error, inspect stderr and input compatibility first. For reward 0, inspect the verifier output and the produced artifact. For a missing reward, determine whether execution, evaluation or artifact collection failed. Change one controlled variable, publish a new Profile version if needed, and submit with a new request ID. Preserve the original Job for comparison.

Evaluate a trained model

Training and evaluation are separate Jobs. Connect them by evaluating the same held-out set before and after training, at the stage that fits your task:

Stage	How	Good for
After training	Inspect saved managed Job checkpoints, then evaluate identical held-out prompts through normal inference APIs	Checkpoint selection and generalization; interactive sampling during training is unsupported
After deployment	Call the endpoint's /sft and /base aliases with identical prompts and settings (Inference)	Before/after comparison on your own scorer
Agent benchmark	el job submit --dataset ... with a Profile	Multi-step agent tasks with verifiers and pass@k

A minimal before/after script for an endpoint. Replace the scorer with your task's verifier and keep the held-out file out of training data:

import json, os
from openai import OpenAI

client = OpenAI(base_url="https://api.envloop.ai/v1/job/YOUR_JOB_ID/Inference",
                api_key=os.environ["ENVLOOP_API_KEY"], timeout=120)
rows = [json.loads(line) for line in open("heldout.jsonl")]  # {"prompt": ..., "answer": ...}

def score(model):
    correct = 0
    for row in rows:
        out = client.chat.completions.create(
            model=model, messages=[{"role": "user", "content": row["prompt"]}],
            max_tokens=256, temperature=0).choices[0].message.content or ""
        correct += out.strip() == row["answer"]
    return correct / len(rows)

for alias in ("YOUR_ENDPOINT_MODEL/base", "YOUR_ENDPOINT_MODEL/sft"):
    print(alias, score(alias))

Run an Agent benchmark on a trained model

Inference endpoints also serve the Anthropic Messages API, so the managed Claude Code Agent can use your trained model directly. Deploy the checkpoint, then publish a Profile whose model is the endpoint:

{
  "schema_version": "envplatform.profile/v2",
  "name": "team/eval-my-model",
  "version": "1",
  "secret_params": ["MODEL_API_KEY"],
  "spec": {
    "params": {
      "AGENT_NAME": "claude-code",
      "AGENT_VERSION": "2.1.285",
      "MODEL_PROTOCOL": "anthropic",
      "MODEL_BASE_URL": "https://api.envloop.ai/v1/job/YOUR_INFERENCE_JOB_ID/Inference",
      "MODEL_NAME": "YOUR_ENDPOINT_MODEL/sft",
      "MODEL_API_KEY": "YOUR_ENVLOOP_KEY",
      "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "4096",
      "CLAUDE_CODE_EFFORT_LEVEL": "medium"
    }
  }
}

el profile create team/eval-my-model 1 --file eval-profile.json
el job submit --dataset DATASET_NAME@VERSION --profile team/eval-my-model@1 --attempts 2

• Set CLAUDE_CODE_EFFORT_LEVEL to an effort the model accepts. Claude Code otherwise requests high, which qwen38-27b@1 rejects with 400 "Unexpected reasoning effort"; Qwen3.8 accepts low, medium and xhigh.
• The managed Claude Code Agent runs only where EnvLoop has enabled it for your project. If submission is rejected, contact EnvLoop support; you can still run Claude Code yourself against the endpoint with ANTHROPIC_BASE_URL and ANTHROPIC_MODEL set as above.
• MODEL_API_KEY is an EnvLoop key that may invoke the endpoint. Prefer an inference-only key restricted to that endpoint; it is stored as a Secret Profile value.
• Publish a second Profile with MODEL_NAME set to YOUR_ENDPOINT_MODEL/base and run the same Dataset, Runner and attempts to compare before and after training on the project leaderboard.
• Claude Code's instructions and tool definitions take a large part of the context. Keep CLAUDE_CODE_MAX_OUTPUT_TOKENS small enough that prompt plus output fit the serving profile's context (32,768 tokens for qwen38-27b@1 and glm53-flash@1); long multi-step tasks can still exceed it.
• The endpoint must be running or able to wake; a cold start delays the first Agent step. Inference time is billed to the endpoint as usual.

Current limits:

• Platform evaluation does not start automatically after training. Record each round yourself: tag training Jobs with agent_run / agent_round (Model Loops) and store the evaluation Job IDs or scores with the same run name.
• A higher score on prompts that were also used for training is not evidence of improvement. Freeze the held-out set and scorer before comparing.