Software Engineering jobs

Member of Technical Staff, Sandbox Infrastructure

San Francisco Tensor CompanySan Francisco, CAFull-timeOn-site

$275k–$315k/yr listed by the employer

Looks real. 1 thing to check before you apply.

On San Francisco Tensor Company’s own careers site (Ashby), checked open today.

  • To check: Posted by the employer 2 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • Checked: On the company's own careers siteAshby board
  • Checked: Checked open todayAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 89 / 100 · Looks good
See all 6 checksShow fewer
How we check

What you’ll do

  • Extend the sandboxing stack to new vendors and accelerators, and build GPU virtualization below the runtime, including gVisor driver- and ioctl-level work.
  • Make sandboxes work on spot capacity through preemption-aware scheduling, checkpointing, and rescheduling.
  • Support multi-GPU and multi-node sandboxes, including NVLink, NVSwitch, and RDMA paths.
  • Own live migration, including socket-preserving migration, and ensure measurement and profiling fidelity and isolation.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • Strong low-level systems engineering background in Linux kernel internals, containers, namespaces, cgroups, syscall interception, or hypervisors
  • Experience in GPU systems engineering: drivers, runtimes, or scheduling on accelerator fleets
  • Comfort with distributed systems failure modes, including preemption, partial failure, and checkpoint/restore
  • Direct experience with gVisor, Firecracker, Kata, QEMU/KVM, or similar (preferred)
  • Direct experience with CRIU, live migration, or connection-preserving failover (preferred)
  • Experience running large fleets on spot or other preemptible capacity (preferred)
  • Security background in isolation boundaries and untrusted code execution (preferred)

Skills

  • Proficiency in Go, C/C++, or Rust
  • Familiarity with NCCL/RCCL, RDMA, InfiniBand, or vendor interconnects (preferred)
  • Familiarity with bare-metal provisioning, hypervisors, or fleet management at scale (preferred)

Other requirements

  • Strong bias toward building solutions yourself when no vendor supports what you need

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Equity
Meaningful equity
Relocation help
Relocation assistance

Full job description

From San Francisco Tensor Company’s careers site

AT SF TENSOR, WE'RE BUILDING THE FUTURE OF HIGH-PERFORMANCE COMPUTE

We firmly believe that the future of AI depends on the unglamorous: rethinking and rebuilding the stack, all the way down. From the hardware underneath it to the compiler targeting it and the cloud running it. Right now those three things fight each other and that friction shows up as a tax on every researcher trying to build something ambitious. We're here to axe that tax and make compute faster, cheaper and more available. When we succeed, compute will be portable enough that "which cloud, which chip" stop being something you worry about.

Show full descriptionShow less

To achieve this, we are building our Kernel Optimizer, which takes code and finds its fastest possible form for whatever vendor and cluster topology you point it at, automatically, as well as the Model Foundry which manages the runs, makes research easier and moves workloads across clouds and chips as prices and availability ship, instead of leaving you locked into whatever vendor you signed with first.

We're backed by Susa Ventures, Y Combinator, along with some great funds and angels including Max Mullen and Paul Graham, as well as founders and executives at Neuralink, Notion and AMD. We're looking for researchers, engineers and organizations who agree with the basic premise: you don't get the next leap in AI without a leap in compute first.

ABOUT THE ROLE

We build the fastest GPU compiler in the world. Most compilers have to preserve correctness at every transform, constraining how far they can search, while we prove correctness at the end instead, allowing us to search a far wider space, with agents, with RL, with anything that works and still guarantee the result. It's why we hold #1 on NVIDIA's own kernel benchmark across hundreds of production kernels.

For our search to work, we need to run an enormous amount of untrusted, freshly generated kernels on real silicon, quickly and safely, which is why we're hiring a Member of Technical Staff for Sandbox Infrastructure to build the layer that makes it possible: a serverless GPU container service across NVIDIA, AMD, TPU and Trainium, at a scale and fidelity nobody sells off the shelf. It needs to run three things: (1) our compiler measures every candidate program inside it, so a noisy or unfair sandbox directly corrupts the reward signal the search learns from, (2) our own post-training runs inside it (e.g., when we RL a model on AMD kernel engineering, every rollout is in a sandbox) and (3) customer workloads that require isolation, such as their RL rollouts, run on it too.

We already do this on NVIDIA and AMD, including having rolled AMD GPU support in gVisor from scratch https://x.com/BenKoska/status/2086211029713965099. The work now is depth and breadth, adding support for more vendors, higher fidelity instrumentation, faster cold starts and more features. A lot of our sandboxes need to run on spot pools without losing works, they can span multiple GPUs and sometimes multiple nodes, they need to survive failure and preemption as well as live-migration (which our stack does while keeping sockets intact so that instances can relocate mid-flight while still streaming data in or out without a hiccup).

WHAT YOU'LL DO

  • You'll extend our sandboxing stack to new vendors and accelerators
  • You'll build and maintain GPU virtualization below the runtime, including gVisor work at the driver and ioctl level
  • You'll make sandboxes first-class citizens on spot capacity, which means preemption-aware scheduling, checkpointing and rescheduling
  • You'll support multi-GPU and multi-node sandboxes, including the interconnect (NVLink, NVSwitch) and RDMA paths (InfiniBand, RoCE) paths those require
  • You'll own live migration end-to-end, including our socket-preserving migration
  • You'll guarantee measurement and profiling fidelity as well as their isolation
  • You'll work directly with the compiler, post-training and kernel teams to ensure their throughput is not capped by sandboxes

WHAT WE'RE LOOKING FOR

  • Someone with strong low-level systems engineering background: Linux kernel internals, containers, namespaces, cgroups, syscall interception or hypervisors
  • Someone with experience in GPU systems engineering: drivers, runtimes or scheduling on accelerator fleets
  • Someone comfortable with distributed systems failure modes: preemption, partial failure, checkpoint/restore and dealing with states you can't afford to loose
  • Someone proficient in Go, C/C++ or Rust
  • Someone with a strong bias toward building the thing yourself when no vendor supports what you need

NICE TO HAVE

  • Someone who's worked directly with gVisor, Firecracker, Kata, QEMU/KVM or similar
  • Someone who's worked directly on CRIU, -live migration or connection-preserving failover work
  • Someone familiar with NCCL/RCCL, RDMA, InfiniBand or vendor interconnects
  • Someone who's run large fleets on spot or other preemptible capacity
  • Someone familiar with bare-metal provisioning, hypervisors or fleet management at scale
  • Someone with a security background in isolation boundaries and untrusted code execution

WHY JOIN US

Most "serverless GPU" products stop at running a container on a GPU. Our's runs arbitrary code across four vendors, on spot capacity, across multiple nodes with migration that keeps live sockets open and timings clean enough to use as an RL reward. It's a hard systems problem and it's directly load-bearing, because the faster and more reliable the sandboxes, the more programs our compiler can search and the faster our models train the same week.

We're a small team operating at frontier scale. We pre-trained foundation models on 4,000 AMD GPUs as a team of three, designed and brought up GB300 NVL72 clusters and designed a TOP500 supercomputer.

We believe that hard problems get solved in person and most of our work happens at our office in San Francisco. We offer relocation assistance and, where possible, we'd like you here as often as possible.

The base salary range for this full-time position is $275,000-$315,000, plus meaningful equity and benefits.

Something wrong with this job? Report it · This flag looks wrong

Similar jobs

Staff Production Engineer

Zscaler · Santa Clara, CA +1 place

$152k–$190k/yrFull-timeHybrid

StrongChecked open todayPosted yesterday

See more like this