Software Engineer: Terminal Benchmark Tasks

About the work

The newest terminal benchmarks no longer ask whether an AI agent can run a few commands. They ask whether it can work inside a specialised domain: understand a real system, operate on its current state and deliver a result that meets several interacting constraints, judged by checks the agent cannot see. We author original tasks at this level, hard enough that today’s strongest models fail most attempts.

How it works

You design a task around a problem from your own field and build the environment it runs in, often a stateful one such as a live database, a multi-service system or a stream processor. You write a short instruction that states the goal rather than the steps, an oracle solution that proves the task is solvable, and a verifier that grades the final state for correctness of meaning, not just format. Each task then goes through automated checks, runs against frontier models to measure its difficulty, and is reviewed by a second expert. You revise it until it passes.

Who this is for

Senior engineers, researchers and specialists with graduate-level knowledge or years of professional practice in a technical field, such as databases, distributed systems, compilers, security, scientific computing, hardware design or financial systems. You enjoy building problems that are genuinely hard rather than merely long, and you can explain exactly where the expert insight lies.

What you’ll do

• Design novel, multi-step terminal tasks grounded in a real technical domain
• Build reproducible Docker environments, including stateful and multi-service setups
• Write concise instructions that fully determine the graded result without listing the steps
• Write oracle solutions and verifiers that check semantic correctness of the final state
• Prove each verifier rejects wrong solutions by testing deliberately broken ones, one rule at a time
• Measure difficulty against frontier models and document the expert crux of each task
• Respond to automated checks and peer review until the task is accepted

Who should apply

• 2+ years of professional experience, with graduate-level or senior expertise in at least one technical domain
• Strong Linux, shell and Docker skills
• Proficiency in Python or a systems language such as C, C++, Go, Rust or Java
• Experience writing tests that catch subtle errors; property-based or metamorphic testing is a plus
• Experience with Terminal-Bench or other AI agent benchmarks is a plus
• Clear written English for instructions, difficulty notes and review responses

Contract & payment terms

• You will be engaged as an independent contractor, not as an employee.
• This is fully remote work, open to applicants in any location.
• Work is project-based. Availability varies, and an application does not guarantee an assignment.
• Pay, scope, review criteria and access are confirmed at onboarding, after you sign a Non-Disclosure Agreement (NDA) and an Independent Contractor Agreement (ICA).
• Please do not share confidential material from a current or former employer or client.