Data & AI jobs

Staff AI Product Analyst

AngiRemote · US onlyFull-time

$165k–$225k/yr listed by the employer

Key facts

Experience
10+ years
Degree
Bachelor's degree

This job is closed.

The employer took it down 5 days ago, after 15 days open. Here are similar open jobs.

  • To check: Posted by the employer 5 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • Checked: On the company's own careers siteAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 89 / 100 · Looks good
See all 5 checksShow fewer
How we check

Similar open jobs

See more like this

What you’ll do

  • Define north-star and guardrail metrics for flagship AI journeys, and align with data science partners on experiment design and causal interpretation.
  • Specify how agents use metrics, data tables, documentation, and APIs, and establish evaluation sets, rubrics, and regression checks for agent outputs.
  • Analyze agent failures and live workflow performance, then prioritize improvements to prompts, retrieval, tools, and data pipelines.
  • Create internal playbooks for agent use, human checkpoints, and communicating uncertainty.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 10+ years of experience across product analytics, data science, or technical product management within a complex, high-scale operating environment
  • 1+ years of direct, hands-on experience working with AI/LLM systems in an applied context, focused on evaluating, trusting, and deploying them responsibly
  • Proven track record designing AI evaluation frameworks, including formal evals, golden datasets, production monitoring, or equivalent rigor in a related data discipline
  • Experience building or evaluating agentic analytics workflows and orchestration frameworks, with familiarity with multi-agent systems (preferred)

Education

  • Bachelor's degree in computer science or a quantitative field, such as Computer Science, Statistics, Mathematics, or a related field

Skills

  • Deep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practices

Other requirements

  • Ability to operate in ambiguity, run structured experiments or pilots, and iterate quickly (preferred)

Benefits

Health insurance
Full medical package
Dental & vision
Full dental and vision package
Retirement plan
401(k) retirement plan with company match
Paid time off
Flexible vacation policy
Equity
Equity package
Bonus
Competitive year-end performance bonus
Other
Pet discount plans

Full job description

From Angi’s careers site

For over 30 years, Angi has powered the future of the home services industry, creating an environment where homeowners and pros benefit from more jobs done well.

For homeowners, our platform is a reliable way to find skilled pros. For pros, we're a reliable business partner who helps them find the winnable work they want, when they want. For employees, we're an amazing place to call home. We can't wait to welcome you.

Angi at a glance:

  • Founded in 1995 as Angie’s List and rebranded in 2021
  • Global company with 9 brands in 8 countries and employees worldwide
  • Homeowners have turned to us for 300 million home projects and counting

ABOUT THE TEAM

Show full descriptionShow less

The Staff AI Product Analyst will define AI-native product analytics for Angi, turning visionary technology into prioritized use cases, robust evaluation disciplines, and a trusted data ecosystem. This role is central to building Angi’s agentic product analytics capability—defining where AI-native analytics creates real leverage, designing the evaluation frameworks that earn organizational trust, and establishing repeatable system standards. As a senior individual contributor, you will have broad influence across the product analytics function and beyond, ensuring that both human teams and AI agents remain aligned on truth, quality, and usefulness.

The ideal candidate is a systems-thinking data expert who bridges the gap between deep AI capabilities and practical business outcomes.

WHAT YOU’LL DO

AI Product Strategy & Metrics Design

  • Flagship Journey Measurement: Partner with AI product, engineering, and domain experts to define north-star and guardrail metrics for flagship AI journeys, focusing on adoption, task completion, quality, and downstream business outcomes.
  • Causal Framing & Experimentation: Align closely with data science partners on test design, interpretation, and causal framing when new AI products ship.
  • Data Layer Systems Thinking: Map how agents interact with metrics, raw tables, documentation, and APIs to ensure requirements for data and analytics engineering are explicit, structured, and testable.

Evaluation, Governance & Backlog Management

  • Eval Framework Architecture: Design and maintain formal evaluation sets, "golden datasets," rubrics, and regression checks for agent outputs to guarantee accuracy, completeness, policy safety, and usefulness.
  • Taxonomy & Failure Mode Analysis: Systematically categorize wrong answers, tool misuse, context gaps, and data edge cases, feeding those insights back into prompts, retrieval (RAG), tools, and data modeling pipelines.
  • Performance Monitoring: Monitor live usage of Agentic AI analytics workflows—tracking success, abandonment, fallback patterns, and human override rates to maintain a prioritized engineering improvement backlog.

Cross-Functional Leadership & Enablement

  • Playbooks & Narrative: Own internal documentation defining when to use agents, required human checkpoints, and how to communicate uncertainty. Support enablement and leadership storytelling around agentic AI analytics.
  • Strategic Collaboration: Partner with business leaders, the Principal Analytics Engineer, and data engineering teams to translate complex AI capabilities into measurable business outcomes.
  • Defining Success: Deliver recurring measurement artifacts (dashboards, reviews, or written readouts) that leadership uses to prioritize roadmaps, while continuously reducing high-severity incidents where agents produce incorrect metrics.

WHO YOU ARE

Minimum Qualifications

  • Bachelor's in computer science or a quantitative field (e.g., Computer Science, Statistics, Mathematics, or related fields).
  • 10+ years of experience across product analytics, data science, or technical product management within a complex, high-scale operating environment.
  • 1+ years of direct, hands-on experience working with AI/LLM systems in an applied context, with a specific focus on evaluating, trusting, and deploying them responsibly.
  • Deep technical proficiency with Python, ML frameworks, data visualization/analytics platforms, and standard data management/governance practices.
  • Proven track record designing AI evaluation frameworks, including formal evals, golden datasets, production monitoring, or equivalent rigor in a related data discipline.
  • Exceptional communication skills with a demonstrated ability to translate highly technical AI insights and tradeoffs into clear narratives for non-technical business stakeholders.

Preferred Qualifications

  • Experience building or evaluating agentic analytics workflows, orchestration frameworks, and familiarity with multi-agent systems.
  • Demonstrated ability to operate fluidly in ambiguity, run structured experiments or pilots, and default to fast action and iteration.
  • A systems-oriented mindset with a healthy skepticism of AI hype, focusing strictly on organizational outcomes and driving technical investments that yield true business value.

Compensation & Benefits

  • The salary band for this position ranges from $165,000 - $225,000 commensurate with experience and performance. Compensation may vary based on factors such as cost of living.
  • This position will be eligible for a competitive year end performance bonus & equity package.
  • Full medical, dental, vision package to fit your needs
  • Flexible vacation policy; work hard and take time when you need it
  • Pet discount plans & retirement plan with company match (401K)
  • The rare opportunity to work with sharp, motivated teammates solving some of the most unique challenges and changing the world

We value diversity

We know that the best ideas come from teams where diverse points of view uncover new solutions to hard problems. We welcome and value individuals who bring diverse life experiences, educational backgrounds, cultures, and work experiences.

Our hiring process may utilize artificial intelligence (AI) tools to assist in candidate screening and assessment. Our AI tools are designed to complement, not replace, human decision-making. If you'd like to learn more, you can find detailed information on our policies here https://docs.google.com/document/d/e/2PACX-1vTzGqmhplnkCRfat79U12LzwVLQZJAhwOm0gEqPUHdrCUD4Q5G5kvvdPKer8pFwHw/pub and here https://docs.google.com/document/d/e/2PACX-1vS6ZoC2omINsowZkNud98a8xjHHM5HTxEiG5DXfPBeoWFauoqroevbpkX4hMtxs0w/pub.

#LI-Remote

Something wrong with this job? Report it · This flag looks wrong