Software Engineer jobs

Software Engineer, Inference Systems

River AI Inc.Palo Alto, CA

$200k–$420k/yr listed by the employer

Above the typical $210k a year for senior Software Engineer jobs in CaliforniaHow we compare

How we compare. Typical is the median yearly pay of 1,288 senior Software Engineer jobs in California that list pay, from the employers' own postings. About means within 5%.

Key facts

Degree
Bachelor's degree or equivalent
US visa
Sponsors US visas

Looks real. 2 things to check before you apply.

On River AI Inc.’s own careers site (Greenhouse), checked open today.

  • To check: Posted by the employer 2 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • To check: Wide pay range: top of the range is 2.1× the bottomAsk what the job pays at your level before you interview.
  • Checked: On the company's own careers siteGreenhouse board
  • Checked: Checked open todayGreenhouse board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundGreenhouse board
  • Trust score 95 / 100 · Strong
See all 7 checksShow fewer
How we check

What you’ll do

  • Build and operate the inference serving runtime, including request scheduling, continuous batching, KV-cache management, distributed execution, and checkpoint loading.
  • Optimize inference for dense and mixture-of-experts models, including fine-tuned models and adapters, balancing latency, throughput, memory use, and fairness.
  • Improve multi-GPU execution, communication, model loading, and RL sampling throughput while preserving model-version consistency.
  • Build streaming, cancellation, and failure-recovery capabilities, and profile bottlenecks using reproducible performance and correctness tests.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • Experience building inference engines or performance-sensitive distributed services
  • Experience extending SGLang, vLLM, TensorRT-LLM, or similar frameworks (preferred)
  • Work on advanced serving techniques such as speculative decoding or disaggregated prefill and decode (preferred)
  • Experience with multi-adapter serving, dynamic checkpoint loading, or RL sampling (preferred)
  • Experience operating model-serving systems under production traffic (preferred)

Education

  • Bachelor’s degree in Computer Science, Computer Engineering, or equivalent practical experience

Skills

  • Strong understanding of transformer inference, GPU memory, concurrency, and networking
  • Proficiency in Python and C++ or Rust
  • Strong debugging and profiling skills across models, runtimes, and services
  • Familiarity with expert parallelism, GPU collectives, and quantized inference (preferred)
  • Familiarity with CUDA graphs, custom kernels, and NVIDIA profiling tools (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Comprehensive health insurance
Dental & vision
Comprehensive dental and vision insurance
Paid time off
Unlimited PTO
Relocation help
Relocation assistance as needed
Other
Visa sponsorship and support with the process for the right candidate

Also: equity

Full job description

From River AI Inc.’s careers site

At River AI, our mission is to create personal AI owned and shaped by each individual. To achieve this, we are rewriting the entire stack from scratch: personal hardware for local inference, bespoke training infrastructure, next-generation UIs, and frontier deep learning research.

Who we are

We are scientists, engineers, and builders from the industry's top tech companies and AI labs. We bring a proven track record of scaling consumer systems for hundreds of millions of users and architecting the pre-training infrastructure behind today's frontier models.

About the Role

Show full descriptionShow less

We are looking for exceptional inference systems engineers to build the engines that serve large models through the River API. Your goal is to deliver fast, reliable inference while making efficient use of GPU compute and memory.

You will take ownership of the serving runtime, from request scheduling and continuous batching to KV-cache management, distributed model execution, and checkpoint loading. Your work will support both customer-facing inference and the sampling workloads that power reinforcement learning.

Working closely with GPU kernel engineers, researchers, and infrastructure engineers, you will bring new models into production and improve their performance across realistic workloads. You will measure success through latency, throughput, reliability, and cost, with careful attention to numerical correctness and model behavior.

What You’ll Do

  • Optimize inference for dense and mixture-of-experts models, including fine-tuned models and adapters.
  • Improve batching, caching, and admission control to balance throughput, latency, memory use, and fairness.
  • Accelerate multi-GPU execution, communication, and model loading while preserving model-version consistency.
  • Improve RL sampling throughput while keeping samples and log probabilities tied to the correct model version.
  • Build reliable streaming, cancellation, and recovery under failures and overload.
  • Profile bottlenecks and validate improvements through reproducible performance and correctness tests.

Skills & Qualifications

Minimum Qualifications:

  • Bachelor’s degree in Computer Science, Computer Engineering, or equivalent practical experience.
  • Experience building inference engines or performance-sensitive distributed services.
  • Strong understanding of transformer inference, GPU memory, concurrency, and networking.
  • Proficiency in Python and C++ or Rust.
  • Strong debugging and profiling skills across models, runtimes, and services.
  • A collaborative mindset and strong ownership of engineering outcomes.

Preferred Qualifications: (We encourage you to apply even if you don't meet all of these)

  • Experience extending SGLang, vLLM, TensorRT-LLM, or similar frameworks.
  • Work on advanced serving techniques, such as speculative decoding or disaggregated prefill and decode.
  • Familiarity with expert parallelism, GPU collectives, and quantized inference.
  • Experience with multi-adapter serving, dynamic checkpoint loading, or RL sampling.
  • Familiarity with CUDA graphs, custom kernels, and NVIDIA profiling tools.
  • Experience operating model-serving systems under production traffic.

Logistics & Benefits

  • Location: Palo Alto, California.
  • Compensation: $200,000–$420,000 USD annual base pay, depending on experience and skills, plus equity.
  • Benefits: Comprehensive health, dental, and vision insurance; unlimited PTO; and relocation assistance as needed.
  • Visa Sponsorship: We sponsor visas and support the process for the right candidate.

Something wrong with this job? Report it · This flag looks wrong

Similar jobs nearby

Software Engineer

ServiceNow · Santa Clara, CA

$126k–$195k/yrFull-time

StrongChecked open todayPosted 3 days ago

Staff Software Engineer

LinkedIn · Mountain View, CA

$156k–$255k/yrFull-timeHybrid

StrongChecked open todayPosted 5 days ago

See more like this