Software Engineer jobs

Member of Technical Staff - ML Systems & Inference

Gimlet LabsSan Francisco, CAFull-timeOn-siteBachelor's degree or equivalent

$150k–$390k/yr listed by the employer

Above the typical $250k a year for staff-level Software Engineer jobs in CaliforniaHow we compare

How we compare. Typical is the median yearly pay of 704 staff-level Software Engineer jobs in California that list pay, from the employers' own postings. About means within 5%.

Looks real. 2 things to check before you apply.

On Gimlet Labs’ own careers site (Ashby), checked open today.

  • To check: Posted by the employer 8 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • To check: Wide pay range: top of the range is 2.6× the bottomAsk what the job pays at your level before you interview.
  • Checked: On the company's own careers siteAshby board
  • Checked: Checked open todayAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 80 / 100 · Looks good
See all 7 checksShow fewer
How we check

What you’ll do

  • Build and improve production inference systems, optimizing latency, throughput, efficiency, and resource utilization.
  • Design execution strategies for batching, scheduling, concurrency, and resource placement.
  • Improve KV cache management, memory efficiency, and system behavior under load.
  • Bring up new models and accelerator architectures and support new inference techniques in production.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • Experience building or operating ML inference or model serving systems
  • Experience with inference runtimes such as TensorRT-LLM, vLLM, or custom serving systems (preferred)
  • Experience with batching, scheduling, and concurrency control in inference systems (preferred)
  • Experience profiling and tuning latency- and throughput-critical systems (preferred)
  • Software development experience in Python and C++ (preferred)

Education

  • Bachelor's degree in a relevant field, or an equivalent combination of education, training, and professional experience

Skills

  • Strong software engineering fundamentals
  • Comfort reasoning about performance, memory usage, and system behavior under load
  • Deep understanding of modern model architectures and attention mechanisms (preferred)
  • Familiarity with KV cache management and memory placement strategies (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Full job description

From Gimlet Labs’ careers site

About us

Gimlet is building the first multi-silicon neocloud designed for fast, efficient AI inference.

We combine large-scale compute infrastructure with an execution platform that partitions AI workloads and maps each stage to the hardware best suited to run it.

We work with foundation labs, hyperscalers, and AI-native companies, giving our team access to technical problems spanning frontier models, production infrastructure, and emerging hardware.

About the role

As a Member of Technical Staff focused on ML Systems, you will build the inference systems that execute models end-to-end in production.

Show full descriptionShow less

You will work on the systems that determine how inference executes across that pipeline: how requests are batched and scheduled, how stages are placed and scaled, how KV cache and intermediate state move between accelerators, and how the system balances latency, throughput, and utilization across different hardware characteristics.

You will work across model serving, batching, scheduling, concurrency, KV cache management, and memory placement. You will help bring up models on novel hardware. You will support new model architectures and inference techniques, improve performance under real production workloads, and partner with compiler, kernel, networking, and distributed systems engineers to optimize the full execution path.

What success looks like

In the first 12-18 months, you will:

  • Improve the latency, throughput, and efficiency of production inference workloads
  • Design execution strategies across batching, scheduling, concurrency, and resource utilization
  • Improve KV cache management, memory efficiency, and execution under load
  • Enable new models, accelerator architectures, and inference techniques to run efficiently in production

You may be a good fit if you have

  • Strong software engineering fundamentals
  • Experience building or operating ML inference or model serving systems
  • Comfort reasoning about performance, memory usage, and system behavior under load
  • Bachelor's degree in a relevant field, or an equivalent combination of education, training, and professional experience.

Strong candidates may also have

  • Experience with inference runtimes such as TensorRT-LLM, vLLM, or custom serving systems
  • Deep understanding of modern model architectures and attention mechanisms
  • Experience with batching, scheduling, and concurrency control in inference systems
  • Familiarity with KV cache management and memory placement strategies
  • Experience profiling and tuning latency- and throughput-critical systems
  • Software development experience in Python and C++

Why join now?

Gimlet is expanding from its core technology into a production neocloud spanning new hardware, customers, and data centers.

  • Solve hard problems.
  • Own meaningful work.
  • Build for production.
  • Help define what’s next.

Agency Policy: Gimlet Labs does not accept unsolicited resumes from recruitment agencies or search firms. Any unsolicited resumes submitted without a signed agreement will be considered the property of Gimlet Labs, and no fees will be paid.

Something wrong with this job? Report it · This flag looks wrong

Similar jobs nearby

See more like this