Lead a team building systems to evaluate tasks across environments, datasets, benchmarks, and specialized workflows.
Set data quality standards and design experiments to assess agent outputs.
Develop methods to validate synthetic data, analyze failure modes, check task mutations, and audit trajectories.
Work with research engineers, domain experts, and data vendors to diagnose quality issues and improve data generation.
Summarised from the posting by AI and can be wrong. The full description is the source.
Requirements
Experience
At least 5 years of research or engineering experience building AI or machine learning data evaluation and quality systems
Experience leading technical teams or projects from problem definition through implementation and iteration
Strong background in AI evaluations, post-training, synthetic data validation, benchmarks, or related evaluation infrastructure
Experience translating research insights into production systems and pipelines, and designing metrics, experiments, and quality processes
Ability to work with subject-matter experts to turn domain judgment into scalable review or data generation systems
Experience in an early-stage startup environment
Skills
Advanced proficiency in Python, Docker, and Linux environments
Full job description
From Clera’s careers site
ABOUT THE ROLE
Lead the research engineering work focused on measuring and improving training data for AI agents. You will shape the strategy and systems used to evaluate data across reinforcement learning environments, synthetic datasets, benchmarks, and domain-specific workflows.
WHAT YOU'LL DO
Lead a team building systems to evaluate large volumes of tasks across environments, datasets, benchmarks, and specialized workflows.
Define data quality strategy by creating quality-control systems, setting standards, and designing experiments to assess agent outputs.
Develop scalable approaches to validate synthetic data, including failure-mode analysis, task mutation checks, and trajectory auditing.
Show full descriptionShow less
Work with research engineers, domain experts, and data vendors to diagnose quality problems and improve data generation.
Turn research findings into production tools, dashboards, validation pipelines, and feedback loops.
Help establish rigorous standards for assessing whether agent training data is useful, and mentor research engineers.
WHAT WE'RE LOOKING FOR
At least 5 years of research or engineering experience building AI or machine learning data evaluation and quality systems.
Experience leading technical teams or projects from problem definition through implementation and iteration.
Advanced proficiency in Python, Docker, and Linux environments.
Strong background in AI evaluations, post-training, synthetic data validation, benchmarks, or related evaluation infrastructure.
Sound judgment about what makes training tasks realistic, learnable, diverse, reliable, and useful.
Experience translating research insights into production systems and pipelines, and designing metrics, experiments, and quality processes.
Ability to work with subject-matter experts to turn domain judgment into scalable review or data generation systems.
Clear written communication, careful attention to detail, and experience in an early-stage startup environment.
COMPENSATION & BENEFITS
Base salary: $130,000 to $225,000 annually. Visa sponsorship is available.