QA & Test Engineer jobs

Senior AI QA Automation Engineer

EncoraRemote · Costa Rica onlyPosted 1 month ago

Pay not listed

Key facts

Experience
5+ years
Degree
Bachelor's degree

This job is closed.

The employer took it down 17 days ago, after 4 days open. Here are similar open jobs.

  • To check: No pay range listedAsk for the pay range before you interview.
  • Checked: On the company's own careers siteGreenhouse board
  • Checked: No reposts foundGreenhouse board
  • Note: Last seen open 20 days agoLive check
  • Trust score 89 / 100 · Looks good

This job is outside the US, so these US-only checks are not applied: pay-transparency law, pay likely required by state law, US visa sponsorship wording.

See all 5 checksShow fewer
How we check

Similar open jobs

Senior QA Engineer

Skylight · Remote · US only

$150k–$170k/yrFull-time

StrongChecked open todayPosted 16 days ago

QA Engineer – Automation

Bright Vision Technologies · Remote · US only

$115k–$130k/yrFull-time

StrongChecked open todayPosted 20 days ago

Senior AI Test Engineer

TestPros · Remote · US only

$50–$80/hrPart-time

StrongChecked open todayPosted 21 days ago

Senior QA/SDET

Vouched · Remote · US only

$180k–$210k/yrFull-time

StrongChecked open todayPosted 15 days ago

See more like this

Summary, requirements and match are for US jobs for now.

Full job description

From Encora’s careers site

Job Title: Senior AI QA Automation Engineer

Key Skills: Python, AI Testing, Test Automation, Playwright, Selenium, API Testing, Azure AI Foundry Evaluations, RAG Evaluation, Prompt Testing, CI/CD, Monitoring

Experience: 5+ YOE.

Location: Costa Rica, Peru, Colombia, Bolivia.

Mode: Remote

We at Coforge are hiring Senior AI QA Automation Engineer (#006fu00000bWsOCAA0-4) with the following skill set.

Key Responsibilities

-

Build automated evaluation frameworks for AI agents, RAG solutions, prompts, tools, workflows, and model outputs, including benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.

-

Show full descriptionShow less

Test AI-specific behaviors including hallucinations, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.

-

Develop agent-powered QA solutions for test-case generation, API and workflow validation, and intelligent UAT, while building UI, API, integration, performance, and end-to-end automation.

-

Implement CI/CD quality gates and continuously monitor production AI quality, drift, reliability, and recurring failure patterns.

Required Skills & Qualifications

-

5+ years of experience in Software Quality Engineering, Test Automation, or Reliability Engineering.

-

Strong Python skills with experience building reusable automated testing frameworks.

-

Hands-on experience with Playwright and/or Selenium.

-

Strong experience with API, UI, integration, regression, end-to-end, and CI/CD-based test automation.

-

Experience with Azure AI Foundry Evaluations, RAG evaluation, and prompt testing.

-

Experience defining evaluation metrics, benchmark datasets, regression suites, and measurable quality thresholds.

-

Knowledge of AI quality concepts such as hallucination testing, grounding, citation accuracy, prompt injection, and agent behavior validation.

-

Experience implementing CI/CD quality gates and production monitoring.

-

Ability to translate business requirements, workflows, and AI behaviors into measurable quality criteria.

-

Bachelor’s degree in Computer Science, Software Engineering, Information Systems, or a related field.

Preferred Skills:

-

Experience testing Generative AI, RAG solutions, copilots, or AI agents.

-

Experience with LLM evaluation frameworks and advanced prompt testing.

-

Experience developing agent-powered test generation, workflow validation, or intelligent UAT assistants.

-

Experience with adversarial testing and Responsible AI testing.

-

Experience monitoring AI quality, drift, reliability, and failure patterns in production environments.

Posted On: 12-10-2026

At Coforge, we hire professionals based solely on their skills and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.

Something wrong with this job? Report it · This flag looks wrong