Lead development of systems for evaluating tasks across reinforcement learning environments, synthetic data, benchmarks, and specialized workflows.
Set data quality strategy and standards, and design experiments to assess agent outputs.
Develop scalable synthetic data validation methods, including failure analysis, task mutation checks, and trajectory audits.
Collaborate with research engineers, domain experts, and data vendors to diagnose quality issues and improve data generation workflows.
Summarised from the posting by AI and can be wrong. The full description is the source.
Requirements
Experience
At least 5 years of research or engineering experience, including building AI or machine learning data evaluation systems
Experience leading technical projects or teams through ambiguous problems, from definition to implementation and iteration
Background in AI research, evaluation, post-training, synthetic data, benchmarks, or data quality infrastructure
Strong understanding of what makes agent training data realistic, learnable, diverse, reliable, and useful
Experience designing metrics, experiments, and quality assurance processes, and translating research into production systems
Experience working with subject-matter experts and converting domain judgment into scalable review or data generation workflows
Skills
Advanced proficiency with Python, Docker, and Linux
Other requirements
On-site in the United States
Benefits
Other
Visa sponsorship is available.
Full job description
From Clera’s careers site
ABOUT THE ROLE
Lead the data quality team at an AI and machine learning company focused on reinforcement learning and agent evaluation. You will shape the strategy and systems used to measure and improve training data, helping ensure it is useful for training agents on real-world tasks.
WHAT YOU'LL DO
Lead development of systems that evaluate tasks across reinforcement learning environments, synthetic data, benchmarks, and specialized workflows.
Define data quality strategy, establish standards, and design experiments to assess agent outputs.
Develop scalable methods to validate synthetic data, including failure analysis, task mutation checks, and trajectory audits.
Show full descriptionShow less
Work with research engineers, domain experts, and data vendors to diagnose quality issues and improve data generation workflows.
Turn research insights into production tools, dashboards, validation pipelines, and feedback loops.
Develop shared standards for identifying training data that is useful beyond superficial correctness.
Mentor research engineers and promote technical rigor, clear communication, and effective execution.
WHAT WE'RE LOOKING FOR
At least 5 years of research or engineering experience, including building AI or machine learning data evaluation systems.
Experience leading technical projects or teams through ambiguous problems, from definition to implementation and iteration.
Advanced proficiency with Python, Docker, and Linux.
Background in AI research, evaluation, post-training, synthetic data, benchmarks, or data quality infrastructure.
Strong understanding of what makes agent training data realistic, learnable, diverse, reliable, and useful.
Experience designing metrics, experiments, and quality assurance processes, and translating research into production systems.
Experience working with subject-matter experts and converting domain judgment into scalable review or data generation workflows.
Clear written communication, strong attention to detail, and ability to work independently in an early-stage startup environment.
COMPENSATION & BENEFITS
Base salary: $130,000 to $225,000 annually. Visa sponsorship is available.