RLTasksby ulam.ai
Sign inAll tasks
Software engineeringM1-Lineage

M1-Lineage

An ML-systems benchmark for repairing stateful training infrastructure without corrupting continuation, optimizer lineage, distributed state, schema history, or durable checkpoints. Eight private stages contain multi-module incidents with semantic defects across resume, shards, surgery, schemas, compression, recovery, transactions, and incident response under attestation, corruption, concurrency, crash, determinism, and resource checks.

Coding agentPrivate verifierRL environment
Version5.0.0
Environments8
RewardScalar · 0–1
DeliveryPrivate ZIP
01 Task contract

A real, versioned RL environment.

An ML-systems benchmark for repairing stateful training infrastructure without corrupting continuation, optimizer lineage, distributed state, schema history, or durable checkpoints. Eight private stages contain multi-module incidents with semantic defects across resume, shards, surgery, schemas, compression, recovery, transactions, and incident response under attestation, corruption, concurrency, crash, determinism, and resource checks.

Agent objective

  • Interact with the supplied stateful environment.
  • Produce verifier-checkable actions or artifacts.
  • Maximise scalar reward under the package contract.

Evaluation

  • 8 verifier-backed environments.
  • Reported reward range 0–1.
  • Package-specific public and private checks.

Delivery boundary

  • Private object stored in Cloudflare R2.
  • Authenticated entitlement required.
  • Short-lived signed URL per download.
02 What you will work on

Distinct environments, one demanding research contract.

Enough detail to understand the intellectual terrain; generated instances, hidden mechanisms, and solution paths remain inside the private package.

EnvironmentMathematical or technical frontierAdaptive research problem
Levels 8–9Full-state resume and flattened shardsPreserve microbatch, optimizer, scheduler, sampler, RNG, AMP, padding, alias, topology, and repartition semantics.
Levels 10–11Model surgery and schema lineageRename, split, merge, add, and delete parameters while migrating optimizer state and executing idempotent migration DAGs.
Levels 12–13Compressed state and shard recoveryNormalize dense, sparse, quantized, and low-precision representations; detect and recover stale, corrupt, duplicated, or parity-recoverable shards.
Level 14Transactional checkpoint publicationHandle concurrent writers, compare-and-swap, locks, crash boundaries, and durable visibility without split-brain state.
Level 15Lineage incident responseSelect the latest valid lineage, reconcile conflicting evidence, recover state, and prove exact training continuation.
03 Why it is interesting

What the supplied evaluation reveals.

A GPT-5.6 Pro clean-agent proxy passed all eight visible suites but earned only three perfect hidden stages, with a 0.5779 official aggregate. Development-family mean was 0.7683 versus 0.5597 on held-out stages, strong evidence that the verifier tests system semantics beyond public fixtures. All receipts verified; owner references passed, starters failed, and 96/96 semantic mutants were killed. The proxy used unsafe-local execution and is not an independently calibrated model run.

We publish aggregate behavior and task structure, while withholding generated instances, hidden labels, exact successful probes, private checks, and solution trajectories.

04 Supplied evaluation

Observed evaluation result.

Shown with its provenance and limitations; it is not a performance guarantee.

i
Methodology matters

Reported result from the evaluation artifact supplied with this package.

Evaluated system / policyGPT-5.6 Pro

As identified by the supplied artifact.

Score0.5779

See the source methodology.

Result artifactIncluded

gpt56pro_v5_score_report.json

Public result record

Machine-readable provenance and the exact displayed metric are available in results.json.

Open result JSON
05 Private delivery

The package stays off the public website.

The paid ZIP will live in a private R2 bucket. Vercel authorizes the buyer and issues a 2–5 minute object URL; R2 serves the bytes directly.

Included with purchase

  • Exact package version 5.0.0
  • Environment and task contracts
  • Verifier or scoring interface
  • Supplied reference/evaluation artifacts
  • Purchase record and licence v1.1
delivery flow
Authenticated buyer + entitlement check
+ private R2 object + 2–5 minute signed URL
= direct, auditable download

Package SHA-256
41b126727d7f2fd0dc187b37e3b043edfb4d7fdf9c805908e3fefbc6afa14c35
06 Licence v1.1

Commercial use, without exclusivity.

One purchase licenses this identified item to one legal organisation for worldwide, perpetual commercial model training, evaluation, research and development. Redistribution and resale of the package are not permitted.

Read the full licenceYotta Content LTD · business customers only
M1-Lineage

Ready to add this environment?

Back to marketplace
Stripe checkout

Business purchase confirmation

Sign in or create an account, then complete secure Stripe Checkout. Access is granted only by the verified payment webhook.