QA & Test Engineer jobs

QA Lead (Automation+Performance)

PhotonUnited StatesFull-time10+ years experience

$38k/yr listed by the employer

Looks real. 1 thing to check before you apply.

On Photon’s own careers site (Oracle Recruiting), checked open today.

  • To check: Posted by the employer 7 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • Checked: On the company's own careers siteOracle Recruiting board
  • Checked: Checked open todayOracle Recruiting board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundOracle Recruiting board
  • Trust score 80 / 100 · Looks good
See all 6 checksShow fewer
How we check

What you’ll do

  • Develop automation strategies and evaluation pipelines for probabilistic AI outputs.
  • Create and maintain golden datasets to benchmark agent performance across prompt and model versions.
  • Test agent tool and API calls, parameters, multi-step workflows, and retrieved data.
  • Monitor prompt and model changes, agent latency, token use, and cost-efficiency in CI/CD.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 10+ years in QA Automation, with a recent focus on AI/ML or LLM-based applications
  • Deep experience with Playwright, Selenium, or Cypress for UI, with a heavy focus on API-level testing and database validation
  • Experience testing Multi-Agent Systems (preferred)
  • Background in Investment Banking or Fintech (preferred)

Skills

  • Expert-level Python skills
  • Experience with testing frameworks like Pytest
  • Familiarity with AI evaluation frameworks such as LangSmith, DeepEval, RAGAS, or Promptfoo
  • Understanding that AI testing often requires scoring rather than a binary pass/fail
  • Ability to work with SQL and JSON to validate data retrieved by agents during RAG processes
  • Knowledge of Prompt Engineering and how it influences software behavior (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Medical benefits for full-time employees
Dental & vision
Dental and vision benefits for full-time employees
Retirement plan
401(k) retirement plan for full-time employees
Paid time off
Paid time off and paid holidays for full-time employees
Bonus
Variable pay and incentives for full-time employees

Full job description

From Photon’s careers site

We are seeking a QA Automation Lead who is ready to move beyond traditional "Pass/Fail" testing. In this role, you will design and build automation frameworks specifically for Agentic AI products. You will focus on evaluating the performance of autonomous agents, ensuring they follow logical reasoning paths, call the correct tools, and provide accurate, safe outputs.

Your mission is to build the "evaluations" (Evals) that define what high-quality AI behavior looks like, moving the needle from unpredictable experiments to production-grade software.

Key Responsibilities

Show full descriptionShow less
  • Non-Deterministic Testing: Develop automation strategies for probabilistic outputs, using model-based evaluation to "test the tester."
  • Building "Eval" Pipelines: Create and maintain "Golden Datasets" to benchmark agent performance across different versions of prompts and models.
  • Tool-Use Validation: Build automated tests to verify that agents call the correct functions/APIs with the right parameters in complex multi-step workflows.
  • Regression Testing for Prompts: Monitor how subtle changes in prompt engineering or model updates (e.g., moving from GPT-4 to Claude 3.5) affect the product’s reliability.
  • Latency & Token Monitoring: Integrate performance testing into the CI/CD pipeline to track agent reasoning time and cost-efficiency.
  • Hallucination Detection: Develop automated checks to identify and report AI hallucinations, bias, or "jailbreak" attempts.
  • Collaboration: Work closely with AI Engineers to translate "vague" business requirements into measurable, automated test cases.

Required Skills & Qualifications

  • Experience: 10+ years in QA Automation, with a recent focus on AI/ML or LLM-based applications.
  • Python Proficiency: Expert-level Python skills (the industry standard for AI testing) and experience with testing frameworks like Pytest.
  • AI Testing Tools: Familiarity with AI evaluation frameworks such as LangSmith, DeepEval, RAGAS, or Promptfoo.
  • API & Backend Testing: Deep experience with Playwright, Selenium, or Cypress for UI, but a heavy focus on API-level testing and database validation.
  • Statistical Mindset: Understanding that AI testing often requires "scoring" (e.g., 85% accuracy) rather than a simple binary pass/fail.
  • Data Skills: Ability to work with SQL and JSON to validate data retrieved by agents during RAG (Retrieval-Augmented Generation) processes.

Preferred Qualifications

  • Experience testing Multi-Agent Systems (where one agent tests another).
  • Knowledge of Prompt Engineering and how it influences software behavior.
  • Background in Investment Banking or Fintech (if applicable) to understand high-stakes data accuracy.

Compensation, Benefits and Duration

Minimum Compensation: USD 38,000

Maximum Compensation: USD 133,000

Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.

Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.

This position is not available for independent contractors

No applications will be considered if received more than 120 days after the date of this post

Something wrong with this job? Report it · This flag looks wrong

Similar jobs

Product Validation Lead

PaceMate™ · Remote · US only

$130k–$160k/yrFull-time

StrongChecked open todayPosted 8 days ago

Senior QA Engineer

Rain Technologies Inc. · New York City, NY

$165k–$185k/yrFull-timeHybrid

StrongChecked open todayPosted 10 days ago

Lab Testing QA Analyst

Sand Cherry Associates · Mount Airy, MD +1 place

$55k–$85k/yrContractHybrid

StrongLast seen open yesterdayPosted 9 days ago

See more like this