Clera (agency)San Francisco, CAFull-timeOn-sitePosted 24 days ago
$200k–$300k/yr listed by the employer
About the typical $242k a year for Machine Learning Engineer jobs in California (all levels)How we compare
How we compare. Typical is the median yearly pay of 670 Machine Learning Engineer jobs in California (all levels) that list pay, from the employers' own postings. About means within 5%.
Key facts
Experience
1+ years
US visa
No US visa sponsorship
This job is closed.
The employer took it down 17 days ago, after 4 days open. Here are similar open jobs.
Checked: On the company's own careers siteAshby board
Design synthetic company environments with realistic email, records, and files for agents to use over extended periods.
Build graders and evaluation systems for assessing advanced AI agents.
Develop benchmarks that correlate with real-world production performance.
Identify and address evaluation failures, including reward hacking and benchmark gaming.
Summarised from the posting by AI and can be wrong. The full description is the source.
Requirements
Experience
Hands-on experience with synthetic data pipelines, benchmarks, evaluations, or post-training
Experience building environments or benchmarks used to train or evaluate agents at advanced AI labs, and implementing graders or evaluation systems
At least 1 to 2 years at a fast-growing or AI-focused startup, with experience building synthetic environments or production-like simulations
Experience simulating software systems that can withstand agent tool calls
High-level competition experience in mathematics, programming, debate, or games (preferred)
Education
Strong academic or equivalent credentials, such as a top-tier university background, founding experience, or participation in a leading startup program
Skills
Ability to identify flaws in established benchmarks and detect and fix reward hacking in evaluation systems
Other requirements
On-site five days a week in San Francisco, California
Benefits
Equity
Meaningful founding equity
Bonus
Commission-style bonus of up to $200,000 or more, tied to the value of delivered environments
Full job description
From Clera’s careers site
ABOUT THE ROLE
Build evaluation environments and benchmarks that measure whether AI agents can perform real company work and meet or exceed human performance. This founding engineering role focuses on agent capability assessment and production readiness, helping establish reliable evidence of agent competence.
WHAT YOU'LL DO
Design synthetic company environments with realistic email, records, and files for agents to use over extended periods.
Create robust graders and evaluation systems for assessing advanced AI agents.
Develop benchmarks that correlate with real-world production performance.
Identify and address evaluation failures, including reward hacking and benchmark gaming.
Show full descriptionShow less
Simulate software tools such as inboxes and CRMs with sufficient fidelity for agent interactions at scale.
WHAT WE'RE LOOKING FOR
Hands-on experience with synthetic data pipelines, benchmarks, evaluations, or post-training; this role is suited to an AI researcher who builds, not a purely infrastructure-focused engineer.
Experience building environments or benchmarks used to train or evaluate agents at advanced AI labs, and implementing graders or evaluation systems.
At least 1 to 2 years at a fast-growing or AI-focused startup, with experience building synthetic environments or production-like simulations.
Ability to identify flaws in established benchmarks and detect and fix reward hacking in evaluation systems.
Experience simulating software systems that can withstand agent tool calls.
Strong academic or equivalent credentials, such as a top-tier university background, founding experience, or participation in a leading startup program.
High-level competition experience in mathematics, programming, debate, or games is valued.
COMPENSATION & BENEFITS
$200,000 to $300,000 annual base salary, with up to $350,000 for exceptional candidates. An additional commission-style bonus of up to $200,000 or more is tied to the value of delivered environments. Meaningful founding equity is included. Visa sponsorship is not available.
LOCATION
On-site, five days a week in San Francisco, California, United States.