Data & AI jobs

Head of Evaluations (Legal AI Benchmarking)

Newcode.aiRemote · US only

Pay not listed

Key facts

Degree
Master's degree or equivalent
US visa
No US visa sponsorship

This job is closed.

The employer took it down 12 days ago, after 8 days open. Here are similar open jobs.

  • To check: Posted by the employer 3 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • To check: No pay range listedAsk for the pay range before you interview.
  • Checked: On the company's own careers siteWorkable board
  • Checked: No reposts foundWorkable board
  • Trust score 85 / 100 · Looks good
See all 5 checksShow fewer
How we check

Similar open jobs

Head of Data

Rocket Lawyer · Remote · US only

$205k–$361k/yr

StrongChecked open todayPosted 7 days ago

See more like this

What you’ll do

  • Design benchmarks for contract drafting, information extraction, legal research, and contract review.
  • Build, source, and maintain datasets.
  • Review and score AI-generated legal text, contract analyses, and statutory interpretations for accuracy and precision.
  • Define evaluation criteria for logical reasoning, citation accuracy, and safe abstention.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • Proven ability to break down complex statutory frameworks and case law into structured, logical data points

Education

  • PhD or Master's in statistics, mathematics, machine learning, or equivalent

Other requirements

  • Authorized to work in the applicable country without employer sponsorship

Full job description

From Newcode.ai’s careers site

Who are we?

At Newcode.ai, we're transforming how law firms and legal professionals harness AI for real-world impact. As part of our collaborative, high-growth team, you'll have the rare opportunity to work side-by-side with visionary founders at the bleeding edge of AI and legal innovation — shaping not just our product, but the future of legal work itself.

Note: We believe in being transparent about what it's like to work at Newcode. As a fast-growing startup, we're building and evolving every day. That means not every process, playbook, or framework is already in place, and priorities can shift quickly.

Show full descriptionShow less

The people who thrive here are comfortable with ambiguity, take ownership, and don't wait for perfect direction. They are resourceful, proactive, and able to "figure it out"—solving problems, creating structure where needed, and helping build the company as they go. If you are good with this then, great! Keep reading to learn more.

Position Overview

We are seeking a highly analytical professional with a strong statistical background to join our Head of Evaluations. In this role, you will design, implement, and scale the testing frameworks used to evaluate our platform. You will ensure our AI products meet the highest standards of legal reasoning, factual accuracy, and regulatory compliance while maintaining a near-zero hallucination rate.

Key Responsibilities

  • Design Legal Benchmarks for: Contract Drafting, Information Extraction, Legal Research, and Contract Review
  • Build, source and maintain relevant datasets
  • Audit AI Output: Review and score complex AI-generated legal text, contract analyses, and statutory interpretations for accuracy and precision and lay out a strategy.
  • Define Evaluation Metrics: Establish clear criteria for grading model performance, specifically focusing on logical reasoning, citation accuracy, and the model's ability to safely abstain from answering.
  • Collaborate with Engineering: Partner directly with Engineering to translate legal errors into actionable technical feedback for model fine-tuning.

Requirements

  • PhD or Masters in statistics, mathematics, machine learning or equivalent
  • Analytical Skills: Proven ability to break down complex statutory frameworks and case law into structured, logical data points.
  • Tech-Savviness: python, panda, numpy, jupiter notebooks and similar statistical models

Visa Sponsorship:

At this time, Newcode is unable to provide visa sponsorship. Candidates must be authorized to work in the applicable country without employer sponsorship.

Something wrong with this job? Report it · This flag looks wrong