Data & AI jobs

Model Test and Measurement Engineer

Defcon AIRemote · US onlyPosted 1 month ago

$150k–$190k/yr listed by the employer

Key facts

Experience
5+ years
Clearance
SECRET clearance required

This job is closed.

The employer took it down 8 days ago, after 11 days open. Here are similar open jobs.

  • Checked: On the company's own careers siteGreenhouse board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundGreenhouse board
  • Note: Last seen open 14 days agoLive check
  • Trust score 100 / 100 · Strong
See all 5 checksShow fewer
How we check

Similar open jobs

Senior Data Engineer

Vericast · Remote · US only

$130k–$150k/yrFull-time

StrongChecked open todayPosted today

Data Engineer

Mindex · Remote · US only

$90k–$140k/yrFull-time

StrongChecked open todayPosted 7 days ago

Senior DataOps Engineer

Voyager Technologies, Inc. · Remote · US only

$140k–$180k/yr

StrongChecked open todayPosted 8 days ago

See more like this

What you’ll do

  • Construct labeled ground-truth datasets and design and run sampled audits.
  • Evaluate scoring and generative AI capabilities across model releases, maintaining version inventories, evaluation records, and rollback triggers.
  • Review model drift and overrides, and produce human-oversight and fairness or disparate-effect evidence.
  • Measure workflow outcomes, including review time, throughput, backlog movement, and override and rework rates.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 5+ years with model validation as a named responsibility
  • Direct experience with ground-truth construction and sampling design
  • Regulated-industry or government model-risk background (preferred)
  • Experience with fairness / disparate-effect testing and human-oversight documentation (preferred)

Skills

  • Strong Python and SQL

Other requirements

  • US citizenship
  • Active US Secret clearance
  • NIST AI RMF or comparable practice (preferred)
  • Active Top Secret clearance (preferred)

Benefits

Health insurance
100% employer-paid comprehensive medical insurance for you and your family
Dental & vision
100% employer-paid dental and vision insurance for you and your family
Paid time off
Unlimited PTO, with manager approval
Parental leave
14 weeks of fully paid parental leave
Equity
Equity package
Bonus
Competitive bonus package
Flexible schedule
Flexible work environment where you manage your work day
Other
Fully remote work environment

Full job description

From Defcon AI’s careers site

ABOUT DEFCON AI

RESILIENCE IN THE FACE OF DISRUPTION. DEFCON AI is an insights company that leverages artificial intelligence, mathematical optimization, data analytics, and software engineering for resilient optimization of complex systems. In today’s dynamically changing world, DEFCON AI’s technology aligns outcomes with operational goals, better decision making, and empowers customers to anticipate assess, and mitigate the impacts of disruptions.

Be the independent voice that keeps the whole program honest — your evaluation is the standard everyone else is held to.

About the Role

Show full descriptionShow less

You'll join the analytics and AI engineering team behind a system that genuinely matters: an AI-assisted platform that brings together records from dozens of disparate data sources, resolves them to the correct individual, highlights what analysts should review first, and provides transparent, explainable recommendations that users can trust. Operating within a secure government cloud environment, the platform tackles complex challenges in AI, data integration, and decision support where quality, trust, and accountability are mission-critical.

As a Model Test & Measurement Engineer, you'll own the evaluation framework that helps ensure those systems perform as intended. You'll build and maintain labeled ground truth datasets, design statistically sound audit and sampling methodologies, measure model performance across releases, and create the evidence packages that support deployment decisions. You'll independently validate both scoring and generative AI capabilities, helping the team understand not only whether a model works, but how confidently its outputs can be trusted.

This is a role with genuine influence. Your assessments will inform release decisions, drive improvement efforts, and provide the objective evidence customers rely on when evaluating system performance. You'll work closely with data scientists, AI engineers, and technical leadership while maintaining the independence needed to provide clear, defensible evaluations. You do not build the models you validate, and you do not approve thresholds or release authorization against your own evidence.

If you're energized by measurement, validation, and making complex AI systems more trustworthy, this is an opportunity to have an outsized impact on both the technology and the mission it supports.

This is a fully remote role with occasional travel to DEFCON AI headquarters, customer sites, and partner facilities as needed.

Key Responsibilities

  • Construct labeled ground truth for model evaluation
  • Design and run sampled audits
  • Gate releases against model version; maintain version inventory, evaluation records, and rollback triggers
  • Run drift and override review
  • Produce human-oversight and fairness / disparate-effect evidence
  • Maintain independence from the build roles: produce evaluation evidence, but do not build the models, approve thresholds, or authorize releases against that evidence
  • Measure workflow improvement: review time, throughput, backlog movement, and override and rework rates
  • Design evaluation-phase QC: sample selection that does not mix populations, the unit of review, and fair comparison when methods or searched sources differ
  • Define the measurement events other teams must emit, and confirm the review workspace emits workflow events from first use

Required Qualifications

  • 5+ years with model validation as a named responsibility, not a side task
  • Direct experience with ground-truth construction and sampling design
  • Strong Python and SQL; comfort working independently from the teams whose models you evaluate
  • US Citizenship Required
  • Active US Secret clearance
  • Elevated personnel security requirements apply to portions of this work and are discussed during screening

Preferred Qualifications

  • Regulated-industry or government model-risk background
  • Experience with fairness / disparate-effect testing and human-oversight documentation
  • NIST AI RMF or comparable practice
  • Active Top Secret clearance

What Success Looks Like

  • Evaluation evidence a customer can rely on, independent of the teams that build the models
  • Releases gated against a clear version and evaluation record
  • Workflow metrics (review time, throughput, backlog, override/rework rates) that give the program an honest read on whether it's working

What We Offer

  • A fully remote, results-based environment
  • Competitive salary, bonus, and equity package
  • 100% employer paid, comprehensive health insurance including medical, dental, and vision for you and your family
  • Unlimited PTO, with your manager's approval
  • Flexible work environment where you manage your work day
  • 14 weeks of fully-paid parental leave

Salary Range: $150,000–$190,000. This represents the typical salary range for this position based on experience, skills, and other factors.

Applicant Data Disclosure

By submitting an application, you acknowledge that Defcon AI uses third-party service providers to facilitate its recruitment and hiring processes. These providers include applicant tracking systems, candidate verification platforms, and fraud detection tools (collectively, "Hiring Platforms"). Your application materials, including your résumé, cover letter, work samples, responses to application questions, and any other information you submit, may be transmitted to and processed by these Hiring Platforms for the following purposes:

  • Managing and administering your application throughout the hiring process;
  • Verifying the accuracy and authenticity of application materials, including by cross-referencing information you provide against publicly available sources and proprietary databases;
  • Identifying indicators of potentially fraudulent, fabricated, or materially misleading application content, including but not limited to discrepancies between submitted materials and publicly available professional profiles, geographic anomalies, and fabricated work histories.

Applications that are flagged through this process as containing indicators of fraud or material misrepresentation may be declined from further consideration. If you have questions about the status of your application or the evaluation process, please contact recruiting@defconai.com.

Defcon AI requires its Hiring Platform providers to process your information solely for the purposes described above and in accordance with applicable law. Your information will be retained only for as long as necessary to fulfill these purposes and any applicable legal obligations, after which it will be deleted in accordance with Defcon AI's data retention policies.

We’re an Equal Opportunity Employer: You’ll receive consideration for employment without regard to race, sex, color, religion, sexual orientation, gender identity, national origin, protected veteran status, or on the basis of disability.

For more information about how your data is used, please refer to our Privacy Policy and Applicant Privacy Notice.

Something wrong with this job? Report it