California jobs

Research Engineer - Model Evaluation & MLOps

SciforiumSan Francisco, CAFull-timeOn-site

$155k–$200k/yr listed by the employer

Key facts

Experience
2+ years
Degree
Master's degree or equivalent

Looks real. 1 thing to check before you apply.

On Sciforium’s own careers site (Ashby), checked open today.

  • To check: Posted by the employer 2 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • Checked: On the company's own careers siteAshby board
  • Checked: Checked open todayAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 95 / 100 · Strong
See all 6 checksShow fewer
How we check

What you’ll do

  • Integrate internal and open-weight language and multimodal models into GPU evaluation and inference environments.
  • Build automated benchmarks for model quality and systems performance, including latency, throughput, and memory usage.
  • Maintain experiment tracking, model registry, and versioning for models, datasets, and evaluation configurations.
  • Automate reproducible workflows from research checkpoints to validated deployments, and monitor for quality or performance regressions.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 2+ years of professional ML or software engineering experience, including work on production ML systems, ML platforms, or MLOps infrastructure
  • Hands-on experience developing, running, or evaluating Transformer-based models using PyTorch, TensorFlow, or JAX
  • Experience implementing model evaluation or benchmarking pipelines, including evaluation metrics, reproducibility, and model comparisons
  • Experience running, benchmarking, and debugging models with one or more GPU inference runtimes, such as vLLM, SGLang, TensorRT-LLM, or equivalent, in containerized cloud or on-premises environments
  • Experience enabling models on AMD GPUs and ROCm (preferred)
  • Contributions to open-source evaluation, model, or ML infrastructure projects (preferred)

Education

  • MS or PhD in Computer Science, Computer Engineering, Machine Learning, or a related technical field, or equivalent practical experience

Skills

  • Solid understanding of Transformer architectures and inference
  • Basic understanding of LLM evaluation metrics and familiarity with experiment tracking and model lifecycle workflows
  • Familiarity with Hugging Face Transformers or similar model libraries (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Medical insurance
Dental & vision
Dental and vision insurance
Retirement plan
401k plan
Paid time off
Flexible time off
Meals
Daily lunch, snacks, and beverages
Other
Competitive salary

Also: equity

Full job description

From Sciforium’s careers site

Sciforium is an AI infrastructure company developing next-generation multimodal AI models and a proprietary, high-efficiency serving platform. Backed by multi-million-dollar funding and direct sponsorship from AMD with hands-on support from AMD engineers the team is scaling rapidly to build the full stack powering frontier AI models and real-time applications.

ABOUT THE ROLE

Show full descriptionShow less

As a Research Engineer focused on Model Evaluation & MLOps, you will build the tools and infrastructure needed to evaluate, deploy, and operate multimodal foundation models reliably. You will rapidly enable Sciforium’s models and the latest open-weight models on GPUs, automate quality and performance benchmarking, and improve the MLOps workflows that connect research experiments to reliable releases.

Key Responsibilities

MODEL ENABLEMENT & AUTOMATED EVALUATION

  • Rapidly integrate new internal and open-weight language and multimodal models into our GPU evaluation and inference environments.
  • Build automated benchmarks for model quality and systems performance, including latency, throughput, and memory usage.
  • Create standardized, reproducible comparisons across Sciforium models, external baselines, and runtime configurations.

MLOPS & MODEL LIFECYCLE

  • Build and maintain experiment tracking, model registry, and versioning for models, datasets, and evaluation configurations.
  • Automate the path from research checkpoints to validated deployments through CI/CD and reproducible workflows.
  • Monitor model quality and systems performance, and diagnose failures or regressions across model and deployment pipelines.

RESEARCH & SYSTEMS COLLABORATION

  • Build reusable tools that help researchers launch evaluations, compare experiments, and reproduce results.
  • Profile end-to-end model workloads and collaborate with distributed systems, inference, and GPU kernel engineers on deeper performance issues.

MUST-HAVES

Candidates may be stronger in some areas than others. We are looking for strong software engineering foundations, hands-on ML systems experience, and depth in at least one of model evaluation, MLOps, or model deployment.

  • Experience: 2+ years of professional ML or software engineering experience, including work on production ML systems, ML platforms, or MLOps infrastructure.
  • Software Engineering: Strong Python and software engineering skills, with experience building reliable production systems.
  • Machine Learning Expertise: Hands-on experience developing, running, or evaluating Transformer-based models (e.g., BERT, ViT, VLMs, or LLMs) using PyTorch, TensorFlow, or JAX, with a solid understanding of their architectures and inference.
  • Model Evaluation & MLOps: Experience implementing model evaluation or benchmarking pipelines, including evaluation metrics, reproducibility, and model comparisons. Basic understanding of LLM evaluation metrics and familiarity with experiment tracking and model lifecycle workflows.
  • GPU Systems: Experience running, benchmarking, and debugging models with one or more GPU inference runtimes, such as vLLM, SGLang, TensorRT-LLM, or equivalent, in containerized cloud or on-premises environments.
  • Communication: Ability to document systems clearly and collaborate across research, infrastructure, and product engineering teams.
  • Education: MS or PhD in Computer Science, Computer Engineering, Machine Learning, or a related technical field, or equivalent practical experience.

NICE-TO-HAVE

  • Familiarity with Hugging Face Transformers or similar model libraries.
  • Experience enabling models on AMD GPUs and ROCm.
  • Contributions to open-source evaluation, model, or ML infrastructure projects.

BENEFITS INCLUDE

  • Medical, dental, and vision insurance
  • 401k plan
  • Daily lunch, snacks, and beverages
  • Flexible time off
  • Competitive salary and equity

EQUAL OPPORTUNITY

Sciforium is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran or disability status.

Something wrong with this job? Report it · This flag looks wrong

Similar jobs

Pre-training Research Engineer

Sciforium · San Francisco, CA

$165k–$225k/yrFull-timeOn-site

Looks goodOpen 2 monthsChecked open todayPosted 2 months ago

GPU Kernel Engineer

Sciforium · San Francisco, CA

$190k–$250k/yrFull-timeOn-site

StrongChecked open todayPosted 15 days ago

Software Engineer, Fullstack

Sciforium · San Francisco, CA

$165k–$210k/yrFull-timeOn-site

StrongOpen 3 monthsChecked open todayPosted 3 months ago

See more like this