QA & Test Engineer jobs

Head of Testing (AI)

Newcode.aiNew York, NY

Pay not listed

Key facts

Experience
5+ years
US visa
No US visa sponsorship

This job is closed.

The employer took it down 12 days ago, after 8 days open. Here are similar open jobs.

  • To check: Posted by the employer 3 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • To check: No pay range listed, though one is likely required in NYAsk for the pay range before you interview. The law here likely requires one.
  • Checked: On the company's own careers siteWorkable board
  • Checked: No reposts foundWorkable board
  • Trust score 80 / 100 · Looks good
See all 5 checksShow fewer
How we check

Similar jobs

QA Manager

Judi Health · Charlotte, NC +2 places

$144k–$180k/yrHybrid

StrongOpen 4 monthsChecked open todayPosted 4 months ago

QA Automation Manager

Impiricus · Remote · US only

$130k–$140k/yrFull-time

StrongChecked open todayPosted 1 month ago

Head of Ecommerce

Skylight · Remote · US only

$200k–$275k/yr

StrongLast seen open yesterdayPosted 1 month ago

See more like this

What you’ll do

  • Develop and execute testing strategy for AI-powered products.
  • Create test frameworks, methodologies, plans, cases, and release documentation.
  • Design processes and metrics for evaluating AI outputs, including accuracy, consistency, hallucination risks, and edge cases.
  • Partner with Engineering and other teams to identify quality gaps, analyze results, and improve products and development processes.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 5+ years of experience in software testing, quality assurance, product quality, or a related technical role
  • Proven experience testing AI-powered products, machine learning systems, or LLM-based applications
  • Experience designing testing frameworks and quality processes from the ground up
  • Experience working cross-functionally with Engineering, Product, and Customer-facing teams
  • Comfortable operating in a fast-paced startup environment with changing priorities
  • Experience building quality teams or functions within a growing organization
  • Experience working with legal technology, enterprise software, or regulated industries (preferred)
  • Experience evaluating LLM applications, AI agents, or generative AI workflows (preferred)
  • Legal domain knowledge or experience working with legal professionals (preferred)

Skills

  • Strong understanding of AI evaluation methodologies and how to measure AI system performance
  • Ability to translate complex technical issues into clear insights and recommendations
  • Familiarity with automated testing frameworks and tooling

Other requirements

  • Authorized to work in the applicable country without employer sponsorship

Full job description

From Newcode.ai’s careers site

Who are we?

At Newcode.ai, we're transforming how law firms and legal professionals harness AI for real-world impact. As part of our collaborative, high-growth team, you'll have the rare opportunity to work side-by-side with visionary founders at the bleeding edge of AI and legal innovation — shaping not just our product, but the future of legal work itself.

Note: We believe in being transparent about what it's like to work at Newcode. As a fast-growing startup, we're building and evolving every day. That means not every process, playbook, or framework is already in place, and priorities can shift quickly.

Show full descriptionShow less

The people who thrive here are comfortable with ambiguity, take ownership, and don't wait for perfect direction. They are resourceful, proactive, and able to "figure it out"—solving problems, creating structure where needed, and helping build the company as they go. If you are good with this then, great! Keep reading to learn more.

Role Overview

As Head of Testing, you will own the vision, strategy, and execution of product testing across Newcode. You will develop the frameworks, processes, and standards needed to evaluate our AI-powered products, identify opportunities for improvement, and ensure we deliver reliable, high-quality solutions to our customers.

This role requires someone who understands both software testing and AI product evaluation — someone who can think critically about how AI systems should be tested, measured, and improved.

You will work closely with Engineering and Legal Engineering to establish a world-class testing function.

Requirements

Responsibilities

  • Develop and execute Newcode’s overall testing strategy across AI-powered products
  • Create testing frameworks and methodologies for evaluating product functionality, usability, accuracy, reliability, and performance
  • Design and implement processes for testing AI outputs, including quality benchmarks, evaluation criteria, and success metrics
  • Build structured test plans, test cases, and documentation to support product releases
  • Identify gaps, risks, and improvement opportunities through rigorous product testing
  • Partner with Engineering team to improve product quality and development processes
  • Establish repeatable workflows for regression testing, user acceptance testing, and release validation
  • Analyze testing results and provide actionable insights to improve product performance
  • Develop best practices for AI evaluation, including assessing accuracy, consistency, hallucination risks, and edge cases
  • Help define quality standards as Newcode scales its products and customer base
  • Build and evolve the testing function, including future team growth and processes

Qualifications

  • 5+ years of experience in software testing, quality assurance, product quality, or a related technical role
  • Proven experience testing AI-powered products, machine learning systems, or LLM-based applications
  • Strong understanding of AI evaluation methodologies and how to measure AI system performance
  • Experience designing testing frameworks and quality processes from the ground up
  • Ability to translate complex technical issues into clear insights and recommendations
  • Strong analytical and problem-solving skills with high attention to detail
  • Experience working cross-functionally with Engineering, Product, and Customer-facing teams
  • Comfortable operating in a fast-paced startup environment with changing priorities
  • Familiarity with automated testing frameworks and tooling
  • Excellent written and verbal communication skills
  • Experience building quality teams or functions within a growing organization

Nice to Have

  • Experience working with legal technology, enterprise software, or regulated industries
  • Experience evaluating LLM applications, AI agents, or generative AI workflows
  • Legal domain knowledge or experience working with legal professionals

Visa Sponsorship:

At this time, Newcode is unable to provide visa sponsorship. Candidates must be authorized to work in the applicable country without employer sponsorship.

Benefits

Why You’ll Love Working with Us

  • Culture of trust and excellence: Be part of a team that values reliability, respect, and initiative, where everyone’s contribution matters and diversity drives creativity.
  • Collaborative environment: Work with talented, driven peers on bleeding-edge legal engineering solutions with real-world impact.
  • Growth and ownership: Shape your role as we grow. Take ownership and drive outcomes from the start of your journey.
  • Inclusive culture of excellence and trust: We reward initiative, respect, integrity, and teamwork, and are committed to building a global, inclusive team. Your drive, curiosity, and ideas matter more than your background

Ready to shape what’s next? Apply today and join us in building a smarter, more efficient world powered by AI.

Something wrong with this job? Report it · This flag looks wrong