Data & AI jobs

Senior Model Evaluation Analyst

Ignite ITSuitland, MDPosted 2 months ago

Pay not listed

Key facts

Experience
8+ years
Degree
Bachelor's degree or equivalent

This job is closed.

The employer took it down 16 days ago, after 4 days open. Here are similar open jobs.

  • To check: No pay range listed, though one is likely required in MDAsk for the pay range before you interview. The law here likely requires one.
  • Checked: On the company's own careers siteWorkable board
  • Checked: No reposts foundWorkable board
  • Note: Last seen open 19 days agoLive check
  • Trust score 89 / 100 · Looks good
See all 5 checksShow fewer
How we check

Similar jobs

Senior System Engineer

Belay Technologies · Columbia, MD

$168k–$240k/yrFull-time

StrongChecked open todayPosted 20 days ago

Senior Analytics Engineer

GliaCell Technologies · Remote · US only

$180k–$260k/yrFull-time

StrongChecked open todayPosted 20 days ago

Senior Data Scientist

Ignite IT · Suitland, MD

Pay not listedHybrid

Looks good2 to checkChecked open todayPosted 2 months ago

Senior AI/ML Engineer

Ignite IT · Suitland, MD

Pay not listedHybrid

Looks good2 to checkChecked open todayPosted 2 months ago

See more like this

What you’ll do

  • Prepare and maintain versioned evaluation datasets, reference labels, and test cases with documented provenance and representative coverage.
  • Run model and coding-method evaluations, calculate performance metrics, compare results with approved baselines, and identify category-level regressions.
  • Investigate difficult or ambiguous cases, prepare examples for subject matter expert adjudication, and track decisions and label corrections.
  • Develop reproducible Python or SQL analyses and preserve the versions of data, code, prompts, models, and configurations behind results.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 8+ years in quantitative analysis, data quality, or model validation, including 3+ years evaluating ML, NLP, or automated classification systems
  • Ability to investigate discrepancies independently, document evidence precisely, and explain findings to technical staff and business or domain experts
  • Experience evaluating LLM outputs, survey write-in coding, or detailed industry, occupation, and demographic classifications (preferred)
  • Experience with annotation or adjudication workflows, federal statistical data, and controlled environments for sensitive information (preferred)

Education

  • Bachelor's degree in statistics, data science, mathematics, computer science, or a related field, or equivalent relevant professional experience

Skills

  • Strong Python and SQL skills for preparing datasets, automating analyses, calculating metrics, and producing reproducible reports
  • Practical understanding of sampling, class imbalance, reference-label quality, data leakage, confusion matrices, and uncertainty in performance estimates

Other requirements

  • Must be able to obtain and maintain applicable Census Special Sworn Status and Personal Identity Verification (PIV) credentials
  • Follow confidentiality requirements and support Census core business hours and coordinated critical processing runs

Benefits

Health insurance
Flexible spending account
Dental & vision
Dental insurance, Vision insurance
Retirement plan
401(k), 401(k) matching
Learning & tuition
Professional development assistance, Tuition reimbursement
Life & disability insurance
Life insurance
Other
Referral program

Also: paid time off

Full job description

From Ignite IT’s careers site

Ignite IT Hub is seeking a senior Model Evaluation Analyst to assess the accuracy and reliability of the U.S. Census Bureau's LLM Autocoder. You will maintain evaluation datasets, run comparisons, investigate coding errors, and produce evidence for Government review. Working with the contract’s Data Scientist and Government subject matter experts (SMEs), you will evaluate enhancements for the American Community Survey (ACS) and expansion to other surveys and fields.

Responsibilities

  • Prepare and maintain versioned evaluation datasets, reference labels, and test cases with documented provenance and representative coverage of coding categories.
Show full descriptionShow less
  • Execute evaluation methods established with the Senior Data Scientist and calculate accuracy, joint coding rates, coverage, and results against agreed error thresholds.
  • Compare candidate models and coding methods with approved baselines; identify material category-level regressions that aggregate results may conceal.
  • Analyze difficult, ambiguous, and underrepresented cases; assemble examples for SME adjudication and track decisions and label corrections.
  • Develop repeatable Python or SQL analyses and evaluation checks, preserving the data, code, prompt, model, and configuration versions behind each result.
  • Maintain test evidence, review-comment disposition, and traceability from requirements and acceptance criteria to evaluated results and deliverables.
  • Prepare clear performance reports, tables, and briefings that explain findings and limitations; support pilot assessments, release reviews, and ongoing monitoring.

Requirements

Required qualifications

  • Bachelor's degree in statistics, data science, mathematics, computer science, or a related field, or equivalent relevant professional experience.
  • 8+ years in quantitative analysis, data quality, or model validation, including 3+ years evaluating ML, NLP, or automated classification systems.
  • Strong Python and SQL skills for preparing datasets, automating analyses, calculating metrics, and producing reproducible reports.
  • Practical understanding of sampling, class imbalance, reference-label quality, data leakage, confusion matrices, and uncertainty in performance estimates.
  • Ability to investigate discrepancies independently, document evidence precisely, and explain findings to technical staff and business or domain experts.

Preferred qualifications

  • Experience evaluating LLM outputs, survey write-in coding, or detailed industry, occupation, and demographic classifications.
  • Experience with annotation or adjudication workflows, federal statistical data, and controlled environments for sensitive information.

Conditions of employment. Position contingent on contract award. Must be able to obtain and maintain applicable Census Special Sworn Status and Personal Identity Verification (PIV) credentials, follow confidentiality requirements, and support Census core business hours and coordinated critical processing runs.

Benefits

Benefits

  • 401(k)
  • 401(k) matching
  • Dental insurance
  • Flexible spending account
  • Health insurance
  • Life insurance
  • Paid time off
  • Professional development assistance
  • Referral program
  • Tuition reimbursement
  • Vision insurance

Something wrong with this job? Report it · This flag looks wrong