Software Engineering jobs

Senior Staff Engineer - Infrastructure Systems

RunwareRemote · US onlyFull-timePosted 7 days ago

Pay not listed

Looks real. 1 thing to check before you apply.

On Runware’s own careers site (Workable), checked open today.

  • To check: No pay range listedAsk for the pay range before you interview.
  • Checked: On the company's own careers siteWorkable board
  • Checked: Checked open todayWorkable board
  • Trust score 89 / 100 · Looks good
See all 4 checksShow fewer
How we check

What you’ll do

  • Own and maintain tested, versioned GPU software baselines, qualify driver and CUDA releases, and roll upgrades across the fleet.
  • Investigate GPU failures to root cause and build acceptance tests, diagnostics, telemetry, and automated remediation.
  • Manage how GPUs are exposed and isolated for workloads across Kubernetes and bare-metal systems, including new GPU generation bring-up.
  • Benchmark inference workloads across GPU generations and software stacks, and use results to inform placement, pricing, and hardware purchases.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 12+ years building and operating systems software, including several years running GPU software in production at scale
  • Experience taking GPU failures to root cause across hardware, firmware, driver, runtime, and application
  • Experience running GPUs under Kubernetes and on bare metal, including device plugins, GPU Operator, MIG, image builds, and provisioning at scale
  • Use AI to accelerate your own output every day and have a clear view of how a small team should operate with these tools
  • Experience building GPU drivers, CUDA libraries, or GPU system software at a GPU vendor, cloud provider, or HPC centre (preferred)
  • Experience with Blackwell-generation hardware (B200, GB200/GB300 NVL72), NVLink and NVSwitch topologies, and liquid-cooled racks (preferred)
  • Multi-GPU and multi-node execution experience with NCCL, GPUDirect, and RDMA over RoCE or InfiniBand (preferred)
  • Firmware qualification across BIOS, BMC, NIC, and GPU, and acceptance testing with OEMs and ODMs (preferred)
  • Experience with inference serving internals such as PyTorch, TensorRT-LLM, vLLM, SGLang, or custom diffusion and video pipelines (preferred)
  • Experience with non-NVIDIA accelerators, such as AMD or custom silicon, and their software stacks (preferred)
  • An open-source project you wrote or maintain that people run in production (preferred)
  • Experience with multi-tenant isolation and confidential computing on shared GPU hardware (preferred)
  • Experience running a 24x7 service with published service levels (preferred)

Skills

  • Hands-on knowledge of the NVIDIA software stack: kernel driver and modules, CUDA runtime and libraries, NVML, DCGM, and Container Toolkit, including version interactions
  • Knowledge of Xid errors, ECC and row remapping, PCIe and NVLink errors, thermal faults, and memory faults
  • Strong Linux internals knowledge, including kernel, device drivers and modules, PCIe, NUMA, IOMMU, cgroups, and containers
  • Experience with perf, eBPF, and ftrace
  • Fluency in Python and at least one of C, C++, Rust, or Go
  • Comfortable using CUDA to write tests or reproduce bugs
  • Working knowledge of how PyTorch, TensorRT, and ONNX Runtime use GPUs, including memory allocation, compilation and caches, and multi-GPU execution

Other requirements

  • Clear technical writing for design documents, postmortems, and ticket updates
  • Comfort owning outcomes in a fast-moving, ambiguous environment and being accountable for results

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Paid time off
Paid vacation, sick days, and public holidays
Parental leave
Paid maternity, paternity, and caregiver leave
Equity
Stock options
Flexible schedule
Flexible hours outside core collaboration blocks
Other
Company retreats twice a year, Remote-first work arrangement; in-person meetings twice a year, Downtime after major release pushes

Full job description

From Runware’s careers site

As Senior Staff Engineer, Infrastructure Systems, you will own the GPU software stack on every Runware node: the firmware, the kernel and NVIDIA driver, the CUDA runtime and libraries, the container runtime, and the way Kubernetes and our inference services see and share a GPU. When a GPU misbehaves anywhere in the fleet, you will find out why and write the software that keeps it from happening again.

Show full descriptionShow less

This is a senior staff-level individual contributor role for someone who knows GPUs from the driver up and still writes code every day. One month that means qualifying a new driver and CUDA release against every model family we serve. The next it is the GPU part of the acceptance suite that every node must pass, a detector that catches a failing GPU from its Xid and ECC history before a customer does, the bring-up of a new GPU generation, or a CUDA error that turns out to start in firmware, a PCIe link or a kernel update. You will work directly with the node operations, Managed Kubernetes, inference performance and Sonic Inference Engine teams, and be the person each of them calls when the problem sits below the framework.

Success means that every GPU in every pod runs a known, tested combination of firmware, driver and CUDA, that upgrades roll across the fleet without failed inferences, that GPU faults are detected, classified and remediated automatically, and that a new GPU generation goes from delivery to serving traffic in days.

What you'll do

  • Own the GPU software baseline. Versioned, reproducible combinations of kernel, NVIDIA driver, firmware, CUDA runtime and libraries (cuDNN, cuBLAS, NCCL, TensorRT) and container toolkit, tested before they roll and rolled across the fleet without downtime.
  • Qualify every driver and CUDA release. Keep the compatibility matrix across GPU generations, frameworks and model families, and catch regressions in correctness, performance and memory use before they reach production.
  • Take GPU failures to root cause: Xid errors, CUDA "unknown error", cuBLAS handle failures, out-of-memory and fragmentation, GPUs that fall off the bus. Separate hardware, firmware, driver, runtime and application causes, and prove which one it is.
  • Build GPU acceptance and diagnostics: the burn-in, DCGM diagnostics, memory and interconnect tests and real-workload tests that gate every node and rack before it serves traffic, and the reports customers receive at handover.
  • Make GPU health visible and self-healing. Pull telemetry from DCGM, the driver, the kernel log and the BMC into one trusted view, classify faults, and automate the drain, reset and replacement paths.
  • Own how GPUs are exposed to workloads: the container runtime, device plugins and the GPU Operator, MIG and sharing policies, and isolation between tenants, on bare metal and on Kubernetes.
  • Bring up new GPU generations and new pods with the hardware and node operations teams: first driver and firmware, NVLink and NVSwitch validation, and the automation that makes the next bring-up take days.
  • Characterize hardware with real workloads. Benchmark image, video, audio and LLM inference on each GPU generation and software stack, and feed the results into placement, pricing and the next purchase.
  • Work with NVIDIA and upstream projects. File driver and firmware bugs with reproducible cases and see them through, track what each release changes, and contribute fixes to the open-source tools we depend on.
  • Use AI to accelerate the work: agents for log and Xid triage, test generation, release qualification and runbook automation, so that the team's output grows faster than its headcount.

Requirements

What we're looking for

  • 12+ years building and operating systems software, with several of those on GPU software running in production at scale.
  • Deep, hands-on knowledge of the NVIDIA software stack: the kernel driver and its modules, the CUDA runtime and libraries, NVML, DCGM and the Container Toolkit, and how their versions interact.
  • A record of taking GPU failures to root cause across hardware, firmware, driver, runtime and application: Xid errors, ECC and row remapping, PCIe and NVLink errors, thermal and memory faults.
  • Strong Linux internals: the kernel, device drivers and modules, PCIe, NUMA and IOMMU, cgroups and containers, and the tools (perf, eBPF, ftrace) that show what a machine is really doing.
  • You write code every day. Fluent in Python and in at least one systems language (C, C++, Rust or Go), and comfortable enough in CUDA to write a test or reproduce a bug yourself.
  • Experience running GPUs under Kubernetes and on bare metal: device plugins, the GPU Operator, MIG, image builds and provisioning at scale.
  • Working knowledge of how inference frameworks use a GPU (PyTorch, TensorRT, ONNX Runtime): memory allocation, compilation and caches, and multi-GPU execution.
  • Clear technical writing. Your design documents, postmortems and ticket updates can be acted on by someone who was not in the room.
  • You use AI to accelerate your own output every day, and you have a clear view of how a small team armed with these tools should operate differently.
  • Comfortable owning outcomes in a fast-moving, ambiguous environment, picking the problem that matters most and being accountable for the result.

Nice to have

  • Time spent building GPU drivers, CUDA libraries or GPU system software at a GPU vendor, a cloud provider or an HPC centre.
  • Experience with Blackwell-generation hardware (B200, GB200/GB300 NVL72), NVLink and NVSwitch topologies, and liquid-cooled racks.
  • Multi-GPU and multi-node execution: NCCL, GPUDirect, and RDMA over RoCE or InfiniBand.
  • Firmware qualification across BIOS, BMC, NIC and GPU, and acceptance testing with OEMs and ODMs.
  • Inference serving internals: PyTorch, TensorRT-LLM, vLLM, SGLang or custom diffusion and video pipelines, and how they load, schedule and fail.
  • Experience with non-NVIDIA accelerators (AMD or custom silicon) and their software stacks.
  • An open-source project you wrote or maintain that people run in production.
  • Multi-tenant isolation and confidential computing on shared GPU hardware.
  • Experience running a 24x7 service with published service levels.

Benefits

Why Runware

You will design the network for an AI supercomputer that is being built right now, with the budget, the hardware, and the mandate to do it properly. Few network architects get to own a fabric from the first pod to the global backbone, with a direct line to the people deciding what the next pod looks like.

We're a remote-first collective, meeting in person twice a year to plan, brainstorm, celebrate wins, and enjoy some face-to-face time. We have core hours for cooperative working and calls, but outside of that your calendar is yours. Work the hours that let you perform at your peak while also building a healthy life.

Our release cycles are fast and intense, but they're followed by real downtime. After big pushes we expect the team to unplug, recharge, and come back ready and stronger than ever for the next leap.

  • Generous paid time off: vacation, sick days, public holidays
  • Meaningful stock options: share in the upside you create
  • Flexible hours: own your schedule outside core collaboration blocks
  • Family leave: paid maternity, paternity, and caregiver time
  • Company retreats: twice-yearly gatherings in inspiring locations

Something wrong with this job? Report it · This flag looks wrong

Similar jobs

See more like this