Engineering Manager jobs

Tech Lead - AI Inference

WEKARemote · US only8+ years experience

Pay not listed

Looks real. 2 things to check before you apply.

On WEKA’s own careers site (Greenhouse), checked open today.

  • To check: Posted by the employer 7 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • To check: No pay range listedAsk for the pay range before you interview.
  • Checked: On the company's own careers siteGreenhouse board
  • Checked: Checked open todayGreenhouse board
  • Checked: No reposts foundGreenhouse board
  • Trust score 75 / 100 · Looks good
See all 6 checksShow fewer
How we check

What you’ll do

  • Lead and grow a team of three developers, providing technical direction, mentoring, and career coaching.
  • Own the core inference infrastructure, including the NVMe Token Warehouse, GDS data paths, and vLLM/LMCache serving stack.
  • Guide design, implementation, and delivery of high-throughput, low-latency LLM inference systems; contribute hands-on to production code.
  • Address inference challenges including disaggregated prefill/decode, off-HBM KV caching, RDMA transport, and GPU memory hierarchies.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 8+ years of professional software engineering
  • Proven experience leading engineers and owning complex production systems
  • Hands-on expertise with LLM serving systems, including KV cache reuse, disaggregated prefill/decode, continuous batching, and multi-tier GPU memory hierarchies (HBM → NVMe)
  • Experience deploying and scaling GPU workloads on Kubernetes
  • Familiarity with RDMA networking, bare-metal GPU clusters (H100/A100), and high-throughput distributed storage
  • Demonstrated ability to mentor and develop engineers, run effective 1:1s, support career growth, and balance technical execution with long-term team health
  • Track record of building reliable, high-throughput systems and continuously improving engineering practices
  • Experience in AI/ML infrastructure or high-performance computing (preferred)

Skills

  • Strong familiarity with vLLM, LMCache, NIXL/NVIDIA Dynamo, or similar frameworks
  • Strong Python and C++ skills
  • Solid grasp of CUDA, GPU memory management, and high-performance I/O, including GPUDirect Storage (GDS), RDMA, and NVMe data paths
  • Rust (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Medical coverage may be included.
Dental & vision
Dental and vision coverage may be included.
Retirement plan
401(k) may be included.
Paid time off
Flexible time off (FTO) and sick time may be included.
Life & disability insurance
Life insurance may be included.
Other
Leave of absence under FMLA and other relevant leave laws may be included.

Full job description

From WEKA’s careers site

We are seeking a Tech Lead to lead our AI Inference team. In this role, you will bridge the gap between complex research and production-grade engineering, while cultivating a high-performing team culture. You will lead and grow a squad of 3 developers, balancing hands-on technical contribution with strong people leadership — setting direction, unblocking your team, and driving execution on high-performance systems that optimize Large Language Model (LLM) serving.

The ideal candidate combines deep technical expertise in inference and scale with the leadership maturity to mentor, motivate, and develop engineers in the evolving ecosystem of serving frameworks like vLLM and LMCache.

Show full descriptionShow less

What You'll Work On

  • Lead & Own: Take end-to-end ownership of AMG's core inference infrastructure — from the NVMe Token Warehouse and GDS data paths to the vLLM/LMCache serving stack — driving technical decisions and delivery outcomes.
  • Technical Direction: Guide a team of engineers through design, implementation, and delivery of high-throughput, low-latency LLM inference systems, setting high standards for code quality, architecture, and reliability.
  • Build at Scale: Stay hands-on across the AMG stack (Python, C++, CUDA, vLLM, NIXL/Dynamo, Kubernetes), contributing directly to production systems while providing technical leadership to the team.
  • Solve Hard Problems: Tackle the real frontier challenges of inference engineering — disaggregated prefill/decode, persistent off-HBM KV caching, RDMA-based transport, and multi-tier GPU memory hierarchies — that define what's possible at scale.
  • Grow People & Teams: Mentor and coach engineers through regular 1:1s, career coaching, and sprint reviews. Foster a culture of ownership, collaboration, and technical excellence within the AMG team.
  • Stay on the Frontier: Track the evolving inference ecosystem, benchmark new tools (SGLang, TRT-LLM, NVIDIA Dynamo), and help the team make timely decisions about when to adopt, build, or pivot.

What We're Looking For

  • Experienced Engineering Leader: 8+ years of professional software engineering, with proven experience leading engineers and owning complex production systems — ideally in AI/ML infrastructure or high-performance computing.
  • Deep AI Inference Background: Hands-on expertise with LLM serving systems — KV cache reuse, disaggregated prefill/decode, continuous batching, and multi-tier GPU memory hierarchies (HBM → NVMe). Strong familiarity with vLLM, LMCache, NIXL/NVIDIA Dynamo, or similar frameworks.
  • Systems Engineering Depth: Strong Python and C++ skills (Rust a plus), with a solid grasp of CUDA, GPU memory management, and high-performance I/O — including GPUDirect Storage (GDS), RDMA, and NVMe data paths.
  • Infrastructure Fluency: Experience deploying and scaling GPU workloads on Kubernetes, with familiarity in RDMA networking, bare-metal GPU clusters (H100/A100), and high-throughput distributed storage.
  • People Leadership: Demonstrated ability to mentor and develop engineers — running effective 1:1s, supporting career growth, and balancing technical execution with long-term team health.

High Bar for Quality: A strong sense of engineering craftsmanship, with a track record of building reliable, high-throughput systems and continuously improving engineering practices.

The WEKA Way:

  • We are Accountable: We take full ownership, always–even when things don’t go as planned. We lead with integrity, show up with responsibility & ownership, and hold ourselves and each other to the highest standards.
  • We are Brave: We question the status quo, push boundaries, and take smart risks when needed. We welcome challenges and embrace debates as opportunities for growth, turning courage into fuel for innovation.
  • We are Collaborative: True collaboration isn’t only about working together. It’s about lifting one another up to succeed collectively. We are team-oriented and communicate with empathy and respect. We challenge each other and conduct positive conflict resolution. We are being transparent about our goals and results. And together, we’re unstoppable.
  • We are Customer Centric: Our customers are at the heart of everything we do. We actively listen and prioritize the success of our customers, and every decision we make is driven by how we can better serve, support, and empower them to succeed. When our customers win, we win.

USA Residents Only: The Total Compensation hiring wage range for this position which the Company reasonably and in good faith expects to pay for the position in the specified geographic areas or locations. Final compensation will be dependent on various factors relevant to the position and candidate such as geographical location, candidate qualifications, certifications, relevant job-related work experience, education, skillset and other relevant business and organizational factors, consistent with applicable law. In addition, the position may include some of the following comprehensive benefits such Medical, Dental, Vision, Life, 401(K), Flexible Time off (FTO), sick time, leave of absence as per the FMLA and other relevant leave laws.

Concerned that you don’t meet every qualification above?

Studies have shown that women and people of color may be less likely to apply for jobs if they don’t meet every qualification specified. At WEKA, we are committed to building a diverse, inclusive and authentic workplace. If you are excited about this position but are concerned that your past work experience doesn’t match up perfectly with the job description, we encourage you to apply anyway – you may be just the right candidate for this or other roles at WEKA.

WEKA is an equal opportunity employer that prohibits discrimination and harassment of any kind. We provide equal opportunities to all employees and applicants for employment without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws. This policy applies to all terms and conditions of employment, including recruiting, hiring, placement, promotion, termination, layoff, recall, transfer, leaves of absence, compensation and training.

Something wrong with this job? Report it · This flag looks wrong

Similar jobs

AI Engineer III

Vericast · Remote · US only

$120k–$140k/yrFull-time

StrongChecked open todayPosted 3 days ago

See more like this