Engineering Manager jobs

Software Engineering Director, Agentic Evaluations

G2Remote · US onlyFull-timePosted 1 month ago

$260k–$320k/yr listed by the employer

Above the typical $220k a year for manager-level remote Engineering Manager jobsHow we compare

How we compare. Typical is the median yearly pay of 224 manager-level remote Engineering Manager jobs that list pay, from the employers' own postings. About means within 5%.

Key facts

Experience
10+ years
Degree
Bachelor's degree

Looks real and checked open today.

On G2’s own careers site (Ashby).

  • Checked: On the company's own careers siteAshby board
  • Checked: Checked open todayAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 89 / 100 · Looks good
See all 5 checksShow fewer
How we check

What you’ll do

  • Scope and build evaluations that test live software-vendor agents and produce credible results.
  • Own the evaluation system across its admin and API surfaces, workflows, and reusable primitives for onboarding software categories.
  • Turn integration onboarding and maintenance processes into repeatable AI skills or agents, and track emerging evaluation methods and result-distribution approaches.
  • Collaborate with data science on benchmarks, mentor and lead engineers, and promote the use of evaluations across agent-oriented products.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 10+ years of professional programming experience in backend or full-stack environments
  • 2+ years of experience directly managing engineers
  • Direct experience creating evaluations against a customer-facing agent, using agent trajectory trace data and rubrics to measure task completion rate, accuracy, correctness, or policy adherence
  • Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications
  • Regular use of coding agent harnesses such as Claude Code, Codex, Opencode, or Pi in the daily development workflow
  • Experience with agent tool use via direct integration or through MCP servers (preferred)
  • Experience using Playwright, browser-use, Chrome DevTools MCP, or similar tools to drive browser user flows (preferred)
  • Experience with durable execution frameworks and workflow solutions such as Temporal, DBOS, Cloudflare Workflows, or Vercel Workflows (preferred)
  • Experience with agent sandboxing using AWS E2B, Daytona, Cloudflare Containers, Vercel Containers, or similar (preferred)

Education

  • BS/BA degree in a related field

Skills

  • Expert-level proficiency in backend development using Python, Java/Kotlin, TypeScript/JavaScript, or Go
  • Strong proficiency in associated backend frameworks such as FastAPI or Node.js
  • Knowledge of STATE-Bench, tau2-bench, or similar benchmark frameworks (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Full job description

From G2’s careers site

About G2 - The Company

G2 is the world's largest and most trusted software marketplace. When you join G2, you’re joining the industry’s leading team that helps businesses reach their peak potential by powering decisions and strategies with trusted insights from real software users.

Show full descriptionShow less

Now, we have joined forces with Capterra https://www.capterra.com/, SoftwareAdvice https://www.softwareadvice.com/, and GetApp https://www.getapp.com/ to create the largest source of online data and software insights to fuel intelligent buying in the age of AI. With 200M+ combined annual visitors and 6M verified reviews, we are now the centralized place to enable software buyers to make better and faster decisions with confidence.

And we are just getting started! We are setting out to transform the global B2B software industry and become the most trusted data foundation for buyers and sellers of software for the age of AI.

Does that sound exciting to you? Come join us as we try to reach our next PEAK!

About G2 - Our People

At G2, everything we are and what we do is grounded in our PEAK values— (Performance + Entrepreneurship + Authenticity + Kindness. Working at G2 means you are part of a value-driven, growing global community that climbs PEAKs together. We cheer for each other’s successes, learn from our mistakes, and support and lean on one another during challenging times. With ambition and entrepreneurial spirit we push each other to take on challenging work, which will help us all to grow and learn.

You will be part of a global, diverse team of smart, dedicated, and kind individuals - each with unique talents, aspirations, and life experiences. At the heart of our community and culture are our people-led ERGs, which celebrate and highlight the diverse identities of our global team. As an organization, we are intentional about our DEI https://company.g2.com/dei and philanthropic work (like our G2 Gives https://company.g2.com/gives program) because it encourages us all to be better people.

ABOUT THE ROLE

Summary of Responsibilities:

As a Software Engineering Director, Agentic Evaluations, your responsibilities will center around delivery of useful and credible evaluations of agents offered by software vendors. You’ll enhance the system that performs validation of agent performance, and lead the core team behind it. You will leverage “agent-first” ways of working, applying agentic engineering techniques to deliver your work to production.

DETAILED RESPONSIBILITIES:

Build and enhance agent evaluations that run against live agents [80%]:

  • Help scope out the feasibility and effort involved in evaluating an agent offered by a software vendor, identifying the most practical path to reaching a credible evaluation
  • Own the full stack of the core evaluation system, from admin & API surface to eval workflow dispatch
  • Design evaluation system primitives that generalize across software verticals, so onboarding a new category benefits from reuse
  • Distill the appropriate and repeatable processes in onboarding and maintaining integrations into repeatable AI skills or agents to increase evaluation velocity
  • Keep tabs on emerging frameworks and techniques for agent evaluation which offer improvements to evaluations, and provide industry-accepted means to distribute eval results
  • Collaborate with data science peers in their pursuit to develop rich and proprietary benchmarks

Lead a group of engineers and be a source of wider influence [20%]:

  • You will facilitate the growth and mentorship of a core team of engineers, and be a source of technical leadership and subject matter expertise in evals for them
  • You will also help socialize the use of evals in agent-oriented products built by the wider organization.

MINIMUM QUALIFICATIONS:

We realize applying for jobs can feel daunting at times. Even if you don’t check all the boxes in the job description, we encourage you to apply anyway.

  • 10+ years of professional programming experience in backend or full-stack environments
  • 2+ years of experience directly managing engineers
  • Expert-level proficiency in backend development using languages like Python, Java/Kotlin, Typescript/Javascript or Go; strong proficiency in associated backend frameworks like FastAPI or Node.js http://node.js
  • Direct experience creating evaluations (evals) against a customer-facing agent, leveraging agent trajectory trace data and rubrics to measure an agent’s task completion rate, accuracy, correctness, or policy adherence
  • Direct experience using frontier models from OpenAI, Anthropic, or Google in LLM-as-a-judge applications
  • Regular use of coding agent harnesses like Claude Code, Codex, Opencode, or Pi as part of the daily development workflow
  • BS/BA degree in related field

WHAT CAN HELP YOUR APPLICATION STAND OUT:

  • Experience with agent tool use via direct integration or as mediated by MCP servers
  • Knowledge of STATE-Bench, tau2-bench or similar benchmark frameworks
  • Experience using Playwright, browser-use, Chrome DevTools MCP or others to drive user flows in the browser
  • Experience with durable execution frameworks & workflow solutions, like Temporal, DBOS, Cloudflare / Vercel Workflows, or similar; or experience with agent sandboxing using AWS E2B, Daytona, Cloudflare / Vercel Containers or similar

Our Commitment to Inclusivity and Diversity

--

How We Use AI Technology in Our Hiring Process G2 incorporates AI-powered technology to enhance our candidate evaluation process. These tools may assist with initial application screening, skills assessment analysis, and identifying candidates whose qualifications align with specific role requirements. While AI technology supports our recruitment workflow, all final hiring decisions remain under human oversight and judgment.

At G2, we are committed to creating an inclusive and diverse environment where people of every background can thrive and feel welcome. We consider applicants without regard to race, color, creed, religion, national origin, genetic information, gender identity or expression, sexual orientation, pregnancy, age, or marital, veteran, or physical or mental disability status. Learn more about our commitments here https://culture.g2.com/commitments.

For job applicants in California, the United Kingdom, and the European Union, please review this applicant privacy notice https://legal.g2.com/applicant-privacy-notice before applying to this job.

Your Choice Matters: If you would prefer that your application be reviewed without AI assistance, you can opt out by entering your email address in the email entry field at the bottom of the Automated Processing Legal Notice. Choosing to opt out will not disadvantage your application in any way—we will ensure your materials receive a thorough manual review by our hiring team. For additional details about how we handle your information throughout the application process, please review G2's Applicant Privacy Notice https://legal.g2.com/applicant-privacy-notice.

Something wrong with this job? Report it · This flag looks wrong

Similar remote jobs

Engineering Manager

Bestow · Remote · US only

$180k–$205k/yrFull-time

StrongChecked open todayPosted 8 days ago

Engineering Manager

Counterpart Health · Remote · US only +1 place

$180k–$230k/yr

StrongChecked open todayPosted 12 days ago

Engineering Manager

Tailscale · Remote · US, Canada +1 place

$206k–$257k/yrFull-time

StrongChecked open todayPosted 16 days ago

See more like this