RLTasksby ulam.ai
Sign inAll tasks
Operations & simulationsSOC-Defender

SOC-Defender

A fully synthetic, offline defensive-security environment where agents triage alerts, investigate identity, endpoint, network, cloud, and change telemetry, preserve evidence, apply targeted containment, update an incident ticket, and submit a structured report. Eight procedural families include genuine attacks and benign lookalikes, with deterministic scoring for containment, evidence, scope, continuity, escalation, and investigation quality.

Decision makingStateful environmentRLVR
Version1.0.0
Environments1
RewardScalar · 0–1
DeliveryPrivate ZIP
01 Task contract

A real, versioned RL environment.

A fully synthetic, offline defensive-security environment where agents triage alerts, investigate identity, endpoint, network, cloud, and change telemetry, preserve evidence, apply targeted containment, update an incident ticket, and submit a structured report. Eight procedural families include genuine attacks and benign lookalikes, with deterministic scoring for containment, evidence, scope, continuity, escalation, and investigation quality.

Agent objective

  • Interact with the supplied stateful environment.
  • Produce verifier-checkable actions or artifacts.
  • Maximise scalar reward under the package contract.

Evaluation

  • Private verifier.
  • Reported reward range 0–1.
  • Package-specific public and private checks.

Delivery boundary

  • Private object stored in Cloudflare R2.
  • Authenticated entitlement required.
  • Short-lived signed URL per download.
02 What you will work on

Distinct environments, one demanding research contract.

Enough detail to understand the intellectual terrain; generated instances, hidden mechanisms, and solution paths remain inside the private package.

EnvironmentMathematical or technical frontierAdaptive research problem
Identity compromiseSessions, credentials, OAuth grantsSeparate stolen sessions and malicious persistence from VPN travel and approved changes, then contain only affected authority.
Endpoint intrusionInfostealers and ransomware precursorsCorrelate process, persistence, memory, network, and user evidence before isolation and escalation.
Cloud and workload abuseService principals and resource accessDistinguish stolen workload credentials from authorized automation and rotate or block without unnecessary outage.
Insider exfiltrationCloud, endpoint, removable media, egressPreserve cross-source evidence, scope the data path, and coordinate targeted containment with legal and business escalation.
Benign lookalikesVPN, travel, maintenanceValidate owners, changes, managed devices, MFA, and clean telemetry, then close accurately with zero disruptive action.
03 Why it is interesting

What the supplied evaluation reveals.

In a blind expert episode, the acting model earned 0.9021 but failed because it treated an authorized change as an incident, adding scope and escalations where none were warranted. A feedback-corrected rerun passed at 0.9927, but is explicitly not the blind score. That gap demonstrates the benchmark's central defensive skill: avoiding disruptive false positives. The package passed 41/41 tests; reference calibration passed 64/64 while no-action, random, and over-containment controls passed none.

We publish aggregate behavior and task structure, while withholding generated instances, hidden labels, exact successful probes, private checks, and solution trajectories.

04 Supplied evaluation

Observed evaluation result.

Shown with its provenance and limitations; it is not a performance guarantee.

i
Methodology matters

The blind seed-17 run exceeded the scalar threshold but failed exact scope and escalation gates. A feedback-corrected, explicitly non-blind rerun scored 0.9927 and passed.

Evaluated system / policyGPT-5.6 Pro-assisted agent

As identified by the supplied artifact.

Blind episode reward0.9021 · failed

1 reported runs.

Result artifactIncluded

soc_defender_evaluation_report.json

Public result record

Machine-readable provenance and the exact displayed metric are available in results.json.

Open result JSON
05 Private delivery

The package stays off the public website.

The paid ZIP will live in a private R2 bucket. Vercel authorizes the buyer and issues a 2–5 minute object URL; R2 serves the bytes directly.

Included with purchase

  • Exact package version 1.0.0
  • Environment and task contracts
  • Verifier or scoring interface
  • Supplied reference/evaluation artifacts
  • Purchase record and licence v1.1
delivery flow
Authenticated buyer + entitlement check
+ private R2 object + 2–5 minute signed URL
= direct, auditable download

Package SHA-256
595d86505a99c54eb2a9f1077a0cfceadeb3b311ed10fdfad4b94eae19874a37
06 Licence v1.1

Commercial use, without exclusivity.

One purchase licenses this identified item to one legal organisation for worldwide, perpetual commercial model training, evaluation, research and development. Redistribution and resale of the package are not permitted.

Read the full licenceYotta Content LTD · business customers only
SOC-Defender

Ready to add this environment?

Back to marketplace
Stripe checkout

Business purchase confirmation

Sign in or create an account, then complete secure Stripe Checkout. Access is granted only by the verified payment webhook.