Software Engineering jobs

Model Behavior Engineer

OvertoneNew York City, NYFull-timeOn-site3+ years experiencePosted 2 days ago

$160k–$190k/yr listed by the employer

Looks real and checked open today.

On Overtone’s own careers site (Ashby).

Trust score ready Nov 27

  • Checked: On the company's own careers siteAshby board
  • Checked: Checked open todayAshby board
  • Checked: Pay listedJob posting
  • Note: Too new for a trust scoreScored after a week of daily checks, from Nov 27
See all 4 checksShow fewer
How we check

What you’ll do

  • Build and iterate prompts for content surfaces and judge rubrics; diagnose production issues and validate prompt fixes with evals.
  • Design and evolve evaluation and content-iteration workflows, including labeling, judge calibration, and production quality monitoring.
  • Decide which content features to measure and set up automated error analysis to surface new failure patterns.
  • Own labeling strategy and tools, using subject-matter experts where needed and human labelers to compare judge assessments with human judgment.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 3+ years in product management or engineering, including shipping LLM-powered features to real users
  • Hands-on experience with evals: writing rubrics, building or running LLM-as-judge setups, and aligning judges against human labels
  • Hands-on prompt engineering experience, including diagnosing output problems, rewriting prompts, and validating fixes with evals
  • Experience with Braintrust or similar eval tooling (preferred)
  • Experience running human labeling programs (preferred)
  • Experience with conversational or voice AI products (preferred)

Skills

  • Comfortable writing code for work such as error-analysis scripts or eval-pipeline changes

Other requirements

  • Eagerness to read lots of AI outputs by hand
  • Work in person at the NYC office in Flatiron 5 days a week
  • Interest in relationships, psychology, or matchmaking (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Medical insurance
Dental & vision
Dental and vision insurance
Retirement plan
401(k)
Paid time off
Paid time off, holidays, and sick leave
Equity
Meaningful stock options
Meals
Daily lunch

Also: commuter benefits

Full job description

From Overtone’s careers site

YOUR ROLE

Your job is to work with our relationship scientist and eval engineer to build and execute an eval strategy that keeps our LLM content high quality at scale.

  • Our relationship scientist defines what good looks like across our content surfaces and drafts golden examples. They also label for LLM judges that need subject matter expertise or calls on tone and voice.
  • Our eval engineer builds the infrastructure to power our content monitoring and iteration: human labeling tools, automated judge calibration tools, Braintrust experiments, and production monitoring dashboards.
Show full descriptionShow less
  • You are the hands-on operator of this system: the eval engineer builds the tools, the relationship scientist defines goldens, and you use both every day to find and fix problems in our content.

The team is early, so you'll have real influence over how this workflow gets built.

WHAT YOU’LL DO

  • Prompt iteration and engineering. Fix straightforward errors yourself: when our judges flag a problem in production content, diagnose the failure, change the prompt, and confirm the fix with evals before it ships. When a fix depends on subject matter expertise, partner with our relationship scientist. You’ll draft prompts for our content surfaces and our judge rubrics.
  • Eval + content engineering strategy. Design our infrastructure for evals and content iteration, including labeling, judge calibration and production quality monitoring and keep evolving it as the product grows.
  • Define what we measure. Work with our relationship scientist to decide which content features to measure, from proactive baseline quality checks to reactive checks driven by errors we find. Set up automated error analysis so new failure patterns surface on their own, not only when someone spots them.
  • Labeling strategy and tools. Own how we label and calibrate our judges, including the tools and workflows that scale our relationship scientist's judgment as efficiently as possible. Make sure they label only where their expertise is needed. Everywhere else, label production outputs and match assessments yourself or through outsourced labelers, so we can check that our judges agree with human judgment.
  • Synthetic testing. Design synthetic tests that tell us whether a new prompt is working before it reaches human testing, and review the results.
  • Production review and prioritization. Read real production outputs across our core surfaces and spot quality issues. Prioritize what to fix first across everything our judges, automated error analysis and manual reviews surface.

WHO YOU ARE

We're open to a very product-minded engineer, or a highly technical product person:

  • 3+ years in product management or engineering, including shipping LLM-powered features to real users
  • Hands-on experience with evals: writing rubrics, building or running LLM-as-judge setups, and aligning judges against human labels
  • Hands-on prompt engineering experience: you can diagnose why an output went wrong, rewrite the prompt, and show with evals that the fix worked
  • Comfortable writing code when the work calls for it, like scripts for error analysis or changes to our eval pipeline
  • A detail-oriented, editorial eye for tone, structure and voice in AI-written content
  • Eagerness to dive in and read lots of AI outputs by hand

Great to have:

  • Experience with Braintrust or similar eval tooling
  • Experience running human labeling programs
  • Experience with conversational or voice AI products
  • Interest in relationships, psychology or matchmaking

COMPENSATION & BENEFITS

Salary of $160k to $190k. The offer within that range depends on experience and skills. This position will receive meaningful stock options with considerable upside potential.

We provide standard benefits such as:

  • Medical, dental and vision insurance
  • 401(k)
  • Commuter benefits, daily lunch and other perks
  • Paid time off, holidays and sick leave

OUR CORE QUALITIES

We look for these qualities in everyone we hire at Overtone.

Curiosity: You seek to understand before you decide. You ask thoughtful questions, test assumptions, and stay open to changing your mind as new information emerges.

Craft: You care deeply about quality and coherence. You simplify complexity, exercise good judgment, and take pride in work that feels intentional, thoughtful, and well-made.

Collaboration: You engage with others honestly and generatively. You challenge ideas with respect, listen deeply, and help synthesize perspectives into clearer, better outcomes.

Care: You take responsibility for people and outcomes. You act with integrity, follow through on commitments, and protect trust—with users, teammates, and the work itself.

ABOUT OVERTONE

Overtone is a modern matchmaking company, from the creator of Hinge. We’re building a voice- and audio-forward service, enabled by AI, grounded in relationship science. There’s no profiles or swiping. We get to know people in their own voice, then make thoughtful introductions, one at a time, to people we think are worth meeting. The service is in beta-testing and not yet open to the public. We’re building a small team of people passionate about relationships and human connection, to work together in person in our NYC office (5 days a week in Flatiron).

EQUAL OPPORTUNITY

Overtone is an equal opportunity employer. We consider all qualified applicants without regard to race, color, religion, creed, sex, pregnancy, sexual orientation, gender identity or expression, national origin, ancestry, age, disability, genetic information, marital or partnership status, caregiver status, veteran or military status, or any other status protected by law. We will consider qualified applicants with arrest and conviction records in a manner consistent with the New York City Fair Chance Act and other applicable law. If you need an accommodation at any point in the application or interview process, email careers@overto.ne and we'll work with you.

Something wrong with this job? Report it

Similar jobs

Software Engineer II

Octane · Irving, TX +1 place

$105k–$135k/yr

StrongChecked open todayPosted 9 days ago

Sr. Software Engineer

NBCUniversal · New York, NY

$120k–$170k/yrFull-timeHybrid

StrongChecked open todayPosted 2 days ago

1759 - DevSecOps Engineer

Sigma Defense · Rome, NY +2 places

$120k–$130k/yrFull-time

StrongChecked open todayPosted 4 days ago

Software Engineer, Product

Ambrook · New York, NY +2 places

$115k–$260k/yr · see postingFull-timeHybrid

Wide pay rangeChecked open todayPosted 5 days ago

See more like this