DevOps, SRE & Cloud Engineer jobs

Member of Technical Staff (AI Infrastructure Engineer)

PerplexitySan Francisco, CA +1 placeFull-timeSan Francisco, CA · Palo Alto, CA

$220k–$405k/yr listed by the employer

Above the typical $166k a year for mid-level DevOps, SRE & Cloud Engineer jobs in CaliforniaHow we compare

How we compare. Typical is the median yearly pay of 59 mid-level DevOps, SRE & Cloud Engineer jobs in California that list pay, from the employers' own postings. About means within 5%.

This job is closed.

The employer took it down 16 days ago, after 5 days open. Here are similar open jobs.

  • To check: Posted by the employer 6 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • Checked: On the company's own careers siteAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 80 / 100 · Looks good
See all 5 checksShow fewer
How we check

Similar open jobs

Infrastructure Engineer

Simple AI · San Francisco, CA

$140k–$250k/yrFull-timeOn-site

StrongChecked open todayPosted 10 days ago

Staff Production Engineer

Zscaler · Santa Clara, CA +1 place

$152k–$190k/yrFull-timeHybrid

StrongChecked open todayPosted 2 days ago

See more like this

What you’ll do

  • Design, deploy, and maintain Kubernetes clusters and Slurm environments for AI training and inference.
  • Build APIs, orchestration systems, resource schedulers, and job management systems for training pipelines and inference services.
  • Benchmark and troubleshoot infrastructure, improve performance and cluster utilization, and implement autoscaling.
  • Develop monitoring and alerting for ML workloads and respond to outages affecting training runs and inference services.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • Experience developing APIs and managing distributed systems for batch and real-time workloads
  • Hands-on experience with Slurm workload management, including job scheduling, resource allocation, and cluster optimization
  • Experience deploying and managing distributed training systems at scale
  • Experience managing GPU clusters and optimizing compute resource utilization
  • Demonstrated experience managing large-scale Kubernetes deployments in production
  • Proven track record with Slurm cluster administration and HPC workload management
  • Experience with TensorFlow or distributed training libraries (preferred)
  • Background in HPC environments, parallel computing, and high-performance networking (preferred)
  • Experience with container registries, image optimization, and multi-stage builds for ML workloads (preferred)
  • 3–5 years of relevant experience in ML systems deployment, focused on cluster orchestration and resource management (preferred)

Skills

  • Expert-level Kubernetes administration and YAML configuration management
  • Proficiency with Slurm job scheduling, resource management, and cluster configuration
  • Python and C++ programming focused on systems and infrastructure automation
  • Hands-on experience with PyTorch in distributed training contexts
  • Strong understanding of networking, storage, and compute resource management for ML workloads
  • Debugging and monitoring skills, including observability tools for containerized environments
  • Experience with Kubernetes custom resource definitions, operators, and cluster management
  • Understanding of container orchestration and distributed systems architecture
  • Familiarity with LLM architecture and training processes, including attention mechanisms and distributed training strategies
  • Experience with Kubernetes operators and custom controllers for ML workloads (preferred)
  • Advanced Slurm administration, including multi-cluster federation and advanced scheduling policies (preferred)
  • Familiarity with GPU cluster management and CUDA optimization (preferred)
  • Knowledge of Terraform, Ansible, and GitOps practices (preferred)

Full job description

From Perplexity’s careers site

We are looking for an AI Infra engineer to join our growing team. We work with Kubernetes, Slurm, Python, C++, PyTorch, and primarily on AWS. As an AI Infrastructure Engineer, you will be partnering closely with our Inference and Research teams to build, deploy, and optimize our large-scale AI training and inference clusters

RESPONSIBILITIES

  • Design, deploy, and maintain scalable Kubernetes clusters for AI model inference and training workloads
  • Manage and optimize Slurm-based HPC environments for distributed training of large language models
  • Develop robust APIs and orchestration systems for both training pipelines and inference services
Show full descriptionShow less
  • Implement resource scheduling and job management systems across heterogeneous compute environments
  • Benchmark system performance, diagnose bottlenecks, and implement improvements across both training and inference infrastructure
  • Build monitoring, alerting, and observability solutions tailored to ML workloads running on Kubernetes and Slurm
  • Respond swiftly to system outages and collaborate across teams to maintain high uptime for critical training runs and inference services
  • Optimize cluster utilization and implement autoscaling strategies for dynamic workload demands

QUALIFICATIONS

  • Strong expertise in Kubernetes administration, including custom resource definitions, operators, and cluster management
  • Hands-on experience with Slurm workload management, including job scheduling, resource allocation, and cluster optimization
  • Experience with deploying and managing distributed training systems at scale
  • Deep understanding of container orchestration and distributed systems architecture
  • High level familiarity with LLM architecture and training processes (Multi-Head Attention, Multi/Grouped-Query, distributed training strategies)
  • Experience managing GPU clusters and optimizing compute resource utilization

REQUIRED SKILLS

  • Expert-level Kubernetes administration and YAML configuration management
  • Proficiency with Slurm job scheduling, resource management, and cluster configuration
  • Python and C++ programming with focus on systems and infrastructure automation
  • Hands-on experience with ML frameworks such as PyTorch in distributed training contexts
  • Strong understanding of networking, storage, and compute resource management for ML workloads
  • Experience developing APIs and managing distributed systems for both batch and real-time workloads
  • Solid debugging and monitoring skills with expertise in observability tools for containerized environments

PREFERRED SKILLS

  • Experience with Kubernetes operators and custom controllers for ML workloads
  • Advanced Slurm administration including multi-cluster federation and advanced scheduling policies
  • Familiarity with GPU cluster management and CUDA optimization
  • Experience with other ML frameworks like TensorFlow or distributed training libraries
  • Background in HPC environments, parallel computing, and high-performance networking
  • Knowledge of infrastructure as code (Terraform, Ansible) and GitOps practices
  • Experience with container registries, image optimization, and multi-stage builds for ML workloads

REQUIRED EXPERIENCE

  • Demonstrated experience managing large-scale Kubernetes deployments in production environments
  • Proven track record with Slurm cluster administration and HPC workload management
  • Previous roles in SRE, DevOps, or Platform Engineering with focus on ML infrastructure
  • Experience supporting both long-running training jobs and high-availability inference services
  • Ideally, 3-5 years of relevant experience in ML systems deployment with specific focus on cluster orchestration and resource management

Something wrong with this job? Report it · This flag looks wrong