Machine Learning Engineer jobs

Member of Technical Staff, Evals

HandshakeRemote · US onlyFull-timePhD or equivalentPosted 1 month ago

$200k–$350k/yr listed by the employer

About the typical $275k a year for staff-level remote Machine Learning Engineer jobsHow we compare

How we compare. Typical is the median yearly pay of 74 staff-level remote Machine Learning Engineer jobs that list pay, from the employers' own postings. About means within 5%.

Looks real and checked open today.

On Handshake’s own careers site (Ashby).

  • Checked: On the company's own careers siteAshby board
  • Checked: Checked open todayAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 89 / 100 · Looks good
See all 5 checksShow fewer
How we check

What you’ll do

  • Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, agents, multimodal models, and reinforcement-learning environments.
  • Develop reward models, verifiers, graders, and feedback systems to measure and improve model behavior.
  • Research evaluation quality and build human-data systems, including expert tasks, annotation methods, data-quality signals, and attribution techniques.
  • Prototype and evaluate systems, diagnose failure modes, and share findings through benchmarks, open-source tools, research, and technical writing.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • Publications at top AI/ML venues such as NeurIPS, ICML, ICLR, or COLM
  • Experience tinkering with agents and shipping high-quality software, benchmarks, or datasets
  • Experience building scalable software and working with agents
  • Comfort operating in an ambiguous, fast-moving environment with substantial ownership

Education

  • PhD in ML/AI, computer science, data science, or a related field, or equivalent research experience in industry

Skills

  • Strong Python skills
  • Strong knowledge of frontier AI, including benchmarks, evaluation techniques, agent harnesses, post-training recipes, and data shapes

Other requirements

  • San Francisco location (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Medical coverage
Dental & vision
Dental and vision coverage
Retirement plan
401(k) match
Paid time off
Flexible PTO, 15 holidays, and 2 flex days
Parental leave
Paid parental leave, fertility benefits, and parental coaching
Equity
Equity in a fast-growing company
Bonus
Referral bonuses
Learning & tuition
$2,000 learning stipend and ongoing development
Wellness & mental health
Mental health support and a $500 wellness stipend
Commuter benefits
Commuting support, free lunch, and gym access at the San Francisco office
Other
Competitive compensation and financial coaching, Team outings

Full job description

From Handshake’s careers site

ABOUT HANDSHAKE

Handshake's mission is to organize expert human knowledge to advance the AI economy. Handshake AI works directly with frontier labs on their most consequential data, evaluation, and post-training challenges, building the systems that turn expert human knowledge into the data and evaluations that make frontier models better.

You will work alongside engineers, researchers, operators, and builders from organizations including Scale AI, Meta, Google, Amazon, xAI, Notion, and Palantir—and help build the systems that make expert human knowledge useful for advancing AI.

THE ROLE

Show full descriptionShow less

We are hiring a Member of Technical Staff, Evals to help define how frontier AI systems are measured, understood, and improved. This is a broad, high-ownership role for researchers who build.

You will partner with AI researchers, domain experts, and customers to develop new benchmarks, reward and verifier systems, agent-evaluation methodologies, and data-quality techniques. You will work on questions at the center of frontier AI progress: what should be measured, how to design evaluations that reflect real capability, how to create high-signal feedback, and how to build the environments and data systems that make those answers actionable.

Early members of the team will have unusual influence over our technical direction, operating culture, and the open-source software, benchmarks, and research products we build. We care more about demonstrated research capability, technical judgment, and a builder's mindset than a specific title, degree, or career path.

Location: San Francisco preferred; we are open to exceptional candidates in other locations.

WHAT YOU'LL DO

  • Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and reinforcement-learning environments.
  • Develop reward models, programmatic verifiers, graders, and other feedback systems that make model behavior measurable and improvable.
  • Research what makes evaluations representative, difficult, reliable, and resistant to shortcutting or reward hacking.
  • Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques.
  • Run fast, rigorous iteration loops: prototype, evaluate, interpret results, diagnose failure modes, and turn learnings into the next benchmark or system.
  • Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing.

WHAT WE'RE LOOKING FOR

  • PhD in ML/AI, computer science, data science, or related fields (or equivalent research experience in industry).
  • Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM.
  • Builders who enjoy tinkering with agents and shipping high-quality software, benchmarks, or datasets (e.g. a strong GitHub profile / OSS contributions, or product portfolio).
  • Strong Python skills, experience building scalable software, working with agents.
  • Strong knowledge of frontier AI: benchmarks, eval techniques, agent harnesses, post-training recipes, data shapes.
  • Comfort operating in an ambiguous, fast-moving environment with substantial ownership.

WHY JOIN

  • Work at the very frontier of AI with most major AI labs, researching some of the most important problems in Data and Evaluations.
  • Publish results and work in public through open-source benchmarks and software, papers, and blogs.
  • Join a rapidly growing company whose data business grew from zero to nearly $1B run rate in a year.
  • Help build an early technical organization where your work shapes the roadmap, standards, and culture.
  • Attend (and publish at) conferences like NeurIPS, ICML, ICLR, COLM.

PERKS

Handshake delivers benefits that help you feel supported—and thrive at work and in life.

The below benefits are for full-time US employees.

🎯 Ownership: Equity in a fast-growing company

💰 Financial Wellness: 401(k) match, competitive compensation, financial coaching

🍼 Family Support: Paid parental leave, fertility benefits, parental coaching

💝 Wellbeing: Medical, dental, and vision, mental health support, $500 wellness stipend

📚 Growth: $2,000 learning stipend, ongoing development

💻 Office: Commuting support, free lunch, and gym in our SF office

🏝 Time Off: Flexible PTO, 15 holidays + 2 flex days

🤝 Connection: Team outings & referral bonuses

Something wrong with this job? Report it · This flag looks wrong

Similar remote jobs

See more like this