Research how consumer LLMs search, reason, and synthesize sources; form and test hypotheses and publish findings.
Analyze product decisions and experiments, including agent evaluations and changes to analytical pipelines.
Design and analyze experiments and build measurement approaches when clean A/B tests are not possible.
Build recurring data pipelines and measurements to track changing model behavior and incorporate durable findings into the product.
Summarised from the posting by AI and can be wrong. The full description is the source.
Requirements
Experience
Experience building in early-stage, nebulous environments
Skills
Expert-level Python
Expert-level SQL
Strong foundations in experiment design, effect estimation, and evaluation metrics
Ability to influence stakeholders with evidence
Full job description
From Petra Labs’ careers site
ABOUT PETRA LABS
Petra Labs is building the attribution and experimentation layer for AI search. We believe the future of commerce will be shaped by AI-driven information retrieval, yet most companies lack the infrastructure to understand and capitalize on this emerging growth channel. Petra Labs uses large-scale simulations, agentic analysis, and deeply embedded integrations to turn AI search from a black box into a measurable growth channel teams can rigorously optimize and allocate spend against.
Raised $5.2M in funding from Work-Bench, Pathlight, and Afore. We work in our SoHo, New York office.
CULTURE
Show full descriptionShow less
We hire a small number of extremely talented people and give them as much ownership and autonomy as possible.
We are truth-seeking. Transparent, direct discourse is encouraged. We challenge existing beliefs in pursuit of always getting better.
We work hard and are unapologetic about the level of effort required to build a generational company.
WHAT YOU'LL DO
Drive our research. Shape the direction of in-house research on how consumer LLMs search, reason, and synthesize sources. Stay on top of frontier AEO/GEO research, form hypotheses, and test them with scientific rigor. Your work will be published.
Set the standard for data-driven decision making. Support the product and engineering teams with the analysis and experiments behind key product decisions, from our agent evals to changes in our core analytical pipelines.
Run experimentation and causal inference. Design and analyze experiments, and build measurement even when clean A/B tests aren't possible.
Turn research into a product. A finding about model behavior is only true for the moment it was measured, and the models change constantly. Build the data pipelines and recurring measurement that keep our research live rather than a snapshot, and push the durable findings into the product.
YOU'LL EXCEL IN THIS ROLE IF YOU
Are an expert in Python and SQL
Have strong statistical and experimentation foundations in experiment design, effect estimation, and evaluation metrics
Have built in early-stage, nebulous environments and can strike the right balance between speed and rigor
Communicate clearly in writing and in person, and can influence stakeholders with evidence