Software Engineer jobs

Research Engineer, Infrastructure, Training Systems

Thinking Machines LabSan Francisco, CAFull-timeOn-site

$400k–$500k/yr listed by the employer

Above the typical $210k a year for Software Engineer jobs in California (all levels)How we compare

How we compare. Typical is the median yearly pay of 3,393 Software Engineer jobs in California (all levels) that list pay, from the employers' own postings. About means within 5%.

Key facts

Degree
Bachelor's degree or equivalent
US visa
Sponsors US visas

Looks real. 2 things to check before you apply.

On Thinking Machines Lab’s own careers site (Ashby), checked open today.

  • To check: Labelled as an evergreen postingWhy: “Application note: This is an evergreen role, and we continuously review applications as opportunities arise.”Some postings collect applications for later. Ask if there is an opening now.
  • To check: Posted by the employer 3 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • Checked: On the company's own careers siteAshby board
  • Checked: Checked open todayAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 70 / 100 · Looks good
See all 7 checksShow fewer
How we check

What you’ll do

  • Design, implement and operate distributed training infrastructure across thousands of GPUs and nodes.
  • Diagnose bottlenecks across compute, memory, networking and storage to improve training throughput, accelerator utilization and time to train.
  • Develop parallelism strategies, runtime components, communication primitives and workload orchestration.
  • Improve reliability of long-running training jobs through observability, checkpointing, failure diagnosis and recovery.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • Hands-on experience building, optimizing or operating distributed model-training infrastructure
  • Depth in at least one relevant area, such as distributed training frameworks, parallelism, GPU communication, low-precision training, runtime systems or training-workload orchestration
  • Rust experience (preferred)
  • Implementing or improving FSDP, ZeRO, tensor, pipeline, expert, context or sequence parallelism (preferred)
  • Improving training MFU, throughput, memory efficiency or cost through profiling, mixed precision, FP4/FP8, kernels or compiler work (preferred)
  • Developing NCCL or equivalent communication runtimes, collective algorithms or topology-aware communication (preferred)
  • Improving distributed checkpointing, restart behavior, fault tolerance or training stability (preferred)
  • Building scheduling or runtime infrastructure for large, long-running model-training workloads (preferred)
  • Contributing substantial features to PyTorch, XLA, Megatron-LM, DeepSpeed or related infrastructure (preferred)

Education

  • Bachelor’s degree in a relevant technical field or equivalent practical experience

Skills

  • Strong engineering skills: write performant, maintainable code and debug complex systems
  • Understanding of deep learning frameworks such as PyTorch or JAX and the systems beneath them
  • Ability to investigate problems across subsystem boundaries and carry a solution through implementation and validation

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Generous health benefits
Dental & vision
Generous dental and vision benefits
Paid time off
Unlimited PTO
Parental leave
Paid parental leave
Relocation help
Relocation support as needed

Full job description

From Thinking Machines Lab’s careers site

ABOUT THINKING MACHINES

The mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it.

ABOUT THE ROLE

Frontier model training depends on systems that can keep large accelerator clusters working efficiently and reliably through runs lasting days or weeks. We’re looking for engineers to build and improve those systems, working closely with researchers on the infrastructure behind large-scale pre-training and mid-training.

Show full descriptionShow less

The work spans the training stack: dividing models across accelerators, improving numerical and hardware efficiency, coordinating distributed workloads, and recovering from failures without losing valuable training time. You might optimize FP4/FP8 training, implement a parallelism strategy, remove a communication bottleneck, or make a large training workload more reliable.

We’re especially interested in two complementary areas:

Distributed Training: Build and optimize the frameworks, runtimes and workflows that make large-scale model training possible. Relevant experience includes FSDP, sharded model and optimizer states, expert or context parallelism, distributed checkpointing, training-job scheduling, and improvements to model FLOPs utilization.

Communication / Collectives: Build and optimize the communication systems that connect accelerators during training. Relevant experience includes NCCL or equivalent runtimes, collective algorithms, topology-aware communication, communication/computation overlap, and fault-tolerant training.

You do not need to specialize in both areas. We value depth in a difficult subsystem, the ability to reason across the training stack, and a track record of turning that understanding into working systems.

Our training systems stack is written in Rust. Prior Rust experience is helpful, but strong systems engineers who are interested in learning it are encouraged to apply.

WHAT YOU’LL DO

  • Design, implement and operate distributed training infrastructure that scales across thousands of GPUs and nodes.
  • Improve training throughput, accelerator utilization and time to train by diagnosing bottlenecks across compute, memory, networking and storage.
  • Develop parallelism strategies, runtime components, communication primitives or workload orchestration that support evolving model architectures.
  • Make long-running training jobs more reliable through better observability, checkpointing, failure diagnosis and recovery.
  • Work closely with researchers to understand new workloads, test systems changes and measure their effect on training performance and correctness.
  • Build reusable, maintainable frameworks and libraries that make experimentation faster and training more reproducible.
  • Share useful findings through documentation, open-source contributions or technical reports.

Your focus will depend on your strengths and the problems the team needs to solve.

WHAT WE’RE LOOKING FOR

  • Strong engineering skills: you write performant, maintainable code and can debug complex systems.
  • Hands-on experience building, optimizing or operating distributed model-training infrastructure, with a clear understanding of your personal contributions.
  • Depth in at least one relevant area, such as distributed training frameworks, parallelism, GPU communication, low-precision training, runtime systems or training-workload orchestration.
  • Understanding of deep learning frameworks such as PyTorch or JAX and the systems beneath them.
  • The ability to investigate problems across subsystem boundaries and independently carry a solution through implementation and validation.
  • A collaborative approach to working with researchers and engineers, including clear communication about evidence, uncertainty and tradeoffs.
  • A bachelor’s degree in a relevant technical field or equivalent practical experience.

We welcome engineers at different career stages. Technical contribution and judgment matter more than a particular title or number of years.

EXPERIENCE THAT COULD BE ESPECIALLY RELEVANT

We encourage you to apply if some, rather than all, of these describe your work:

  • Implementing or improving FSDP, ZeRO, tensor, pipeline, expert, context or sequence parallelism.
  • Improving training MFU, throughput, memory efficiency or cost through profiling, mixed precision, FP4/FP8, kernels or compiler work.
  • Developing NCCL or equivalent communication runtimes, collective algorithms, or topology-aware communication.
  • Improving distributed checkpointing, restart behavior, fault tolerance or training stability.
  • Building scheduling or runtime infrastructure for large, long-running model-training workloads.
  • Contributing substantial features to projects such as PyTorch, XLA, Megatron-LM, DeepSpeed or related infrastructure.

We’re interested in what you built, the scale and constraints you worked with, how you diagnosed problems, and what improved as a result.

LOGISTICS

Location: San Francisco, California.

Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $400,000 - $500,000 USD.

Visa sponsorship: We sponsor visas. While we can’t guarantee success for every candidate or role, if you’re the right fit, we’re committed to working through the visa process together.

Benefits: Generous health, dental and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.

Application note: This is an evergreen role, and we continuously review applications as opportunities arise. There may not always be an immediate opening aligned with your experience. Please avoid applying more than once every six months; you are welcome to apply separately to a specific team or project opening.

Something wrong with this job? Report it · This flag looks wrong

Similar jobs nearby

Software Engineer

Simple AI · San Francisco, CA

$130k–$280k/yr · see postingFull-timeOn-site

Wide pay rangeChecked open todayPosted 9 days ago

Software Engineer, Product

Ambrook · New York, NY +2 places

$115k–$260k/yr · see postingFull-timeHybrid

Wide pay rangeChecked open todayPosted 4 days ago

Software Engineer, Robotics

Integrated Biosciences, Inc. · Redwood City, CA

$140k–$230k/yrOn-site

StrongChecked open todayPosted 7 days ago

See more like this