Machine Learning Engineer jobs

Senior Machine Learning Engineer, Autonomous Defense

Horizon3Remote · US onlyFull-time

$211k–$249k/yr listed by the employer

Above the typical $196k a year for senior remote Machine Learning Engineer jobsHow we compare

How we compare. Typical is the median yearly pay of 111 senior remote Machine Learning Engineer jobs that list pay, from the employers' own postings. About means within 5%.

Key facts

Experience
5+ years
Degree
Bachelor's degree or equivalent

This job is closed.

The employer took it down 4 days ago, after 17 days open. Here are similar open jobs.

  • To check: Posted by the employer 2 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • Checked: On the company's own careers siteAshby board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundAshby board
  • Trust score 89 / 100 · Looks good
See all 5 checksShow fewer
How we check

Similar open jobs

See more like this

What you’ll do

  • Build and operate training, post-training, data preparation, and experiment-tracking pipelines for models.
  • Develop model inference and serving infrastructure, including provider routing, fallback, batching, caching, and regional pinning.
  • Manage model releases with versioned configurations, shadow and canary deployments, rollback, monitoring, and per-tenant cost controls.
  • Build retrieval and embedding pipelines and product features in ETL and GraphQL, and collaborate with researchers to productionize models.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 5+ years of professional software engineering experience
  • Demonstrated experience taking ML- or LLM-backed systems from prototype to production and operating them
  • Hands-on experience with ML pipelines and tooling, including training or fine-tuning workflows, experiment tracking, artifact and model registries, and reproducible data preparation
  • Experience building and operating production inference or model-serving infrastructure, including latency and cost optimization
  • Experience building applications on cloud computing platforms such as AWS, Azure, or GCP
  • Experience with container technologies such as Docker and Kubernetes
  • Experience operationalizing LLM or agentic systems (preferred)
  • Hands-on post-training experience with supervised fine-tuning, distillation, preference optimization, or RL, including infrastructure for the runs (preferred)
  • Experience with model gateways or multi-provider routing, and self-hosted or customer-hosted inference (preferred)
  • Experience with GPU infrastructure, quantization, or inference optimization (preferred)
  • Experience with relational databases such as PostgreSQL, graph databases such as Neo4j, and GraphQL backends (preferred)
  • Experience with Datadog, Prometheus, Grafana, and distributed tracing across model calls (preferred)
  • Experience shipping ML into regulated, air-gapped, or customer-controlled environments, or under compliance regimes such as FedRAMP (preferred)

Education

  • Bachelor's degree in Computer Science, Computer Engineering, or related field, or equivalent practical experience

Skills

  • Strong production Python
  • Solid proficiency in SQL and experience with production data pipelines
  • Experience with vLLM, TGI, Bedrock, or similar inference systems (preferred)

Benefits

Health insurance
Health insurance for you and your family
Dental & vision
Vision and dental insurance for you and your family
Paid time off
Flexible vacation policy
Parental leave
Generous parental leave
Equity
Stock options; all full-time roles are eligible for an equity package
Other
Remote and hybrid work models depending on role and location

Full job description

From Horizon3’s careers site

Get to Know Us

Horizon3 is a fast-growing, remote cybersecurity company dedicated to the mission of enabling organizations to proactively find and fix and verify exploitable attack vectors before criminals exploit them. Our flagship product, the NodeZeroTM platform, delivers production-safe autonomous pentests and other key assessment operations that scale across the largest internal, external, cloud, and hybrid cloud environments. NodeZero has been adopted by organizations of all sizes, from small educational institutions to government agencies and Global 100 enterprises. It is used by ITOps/SecOps teams, consulting pentesters, and MSSPs and MSPs.

Show full descriptionShow less

We are a fusion of former U.S. Special Operations cyber operators, startup engineers, and formerly frustrated cybersecurity practitioners. We're committed to helping solve our common security problems: ineffective security tools, false positives resulting in alert fatigue, blind spots, "checkbox” security culture, cybersecurity skills shortage, and the long lead time and expense of hiring outside consultants. Collectively, we are a team of learn it alls, committed to a culture of respect, collaboration, ownership, and results.

What You'll Do

As a Senior Machine Learning Engineer on the Autonomous Defense team, you'll be the person who gets our defensive models into production and keeps them there. Our AI researchers define how NodeZero's agents should reason. You build the pipelines, serve infrastructure, and release machinery that turns that research into something running against thousands of customer tenants every day.

This is the seam where most AI products fail. A model that performs well in a notebook is not a capability. It becomes one when there is a reproducible training pipeline, an evaluation suite that runs in CI, a versioned release path with canaries and rollback, monitoring that catches regressions before customers do, and a cost and latency profile the business can afford. That is your work.

You'll work directly with our AI researchers and closely with backend and infrastructure engineers. You are not being hired to do research, and you are not being hired to do generic platform work. You own the path from model to production.

Responsibilities

  • Build and own the training and post-training pipelines — data preparation, fine-tuning and preference optimization runs, experiment tracking, artifact management, and reproducibility.
  • Build the inference and serving layer: model gateway with provider routing and fallback, regional pinning for data residency, batching, and caching.
  • Own the release path for model-layer artifacts: version prompts, model selections, and tool definitions as deployable config; run shadow and canary deployments by tenant; make rollback fast and boring.
  • Build monitoring for the model layer — behavioral drift, regression detection, output quality signals, latency, and per-tenant token and cost accounting with budget enforcement.
  • Build the data and context pipelines that feed inference, including retrieval and embedding infrastructure over attack path, configuration, and remediation data, with tenant isolation enforced end to end.
  • Optimize cost and latency across the inference path, and make the tradeoffs visible so product decisions are made with real numbers.
  • Develop core product features in ETL and GraphQL where model outputs, run history, and evaluation results need to reach the product and internal tooling.
  • Partner with researchers to move prototypes into production, and feed production constraints and failure data back into research direction.

Required Education / Experience

  • Bachelor's Degree in Computer Science, Computer Engineering or related field, or equivalent practical experience.
  • 5+ yrs professional software engineering experience, with strong production Python.
  • Demonstrated experience taking ML or LLM-backed systems from prototype to production and operating them.
  • Hands-on experience with ML pipelines and tooling: training or fine-tuning workflows, experiment tracking, artifact and model registries, and reproducible data preparation.
  • Experience building and operating inference or model-serving infrastructure in production, including latency and cost optimization.
  • Experience building applications on cloud computing platforms such as AWS, Azure, GCP, using container technologies such as Docker and Kubernetes.
  • Solid proficiency in SQL and experience with production data pipelines.

What Sets You Apart

  • Experience operationalizing LLM or agentic systems specifically.
  • Hands-on post-training experience: supervised fine-tuning, distillation, preference optimization, or RL, including the infrastructure around the runs.
  • Experience with model gateways or multi-provider routing, and with self-hosted or customer-hosted inference (vLLM, TGI, Bedrock, or similar).
  • Experience with GPU infrastructure, quantization, or inference optimization.
  • Experience with database architectures including relational (PostgreSQL) and graph (Neo4j), and with GraphQL backends.
  • Experience with observability tooling (Datadog, Prometheus, Grafana) and distributed tracing, including tracing across model calls.
  • Experience shipping ML into regulated, air-gapped, or customer-controlled environments, or under compliance regimes like FedRAMP.

Perks of Horizon3

  • Inclusive Team: We value diversity and promote an inclusive culture where everyone can thrive.
  • Growth Opportunities: Be part of a dynamic and growing team with numerous career development opportunities.
  • Innovative Culture: Work in a collaborative environment that encourages creativity and out-of-the-box thinking.
  • Hybrid & Remote Work: We embrace a mix of remote and hybrid work models depending on role and location, including our Chicago office, where some roles require regular in-office presence.
  • Competitive Compensation: We offer competitive salary, equity and benefits. Our benefits include health, vision & dental insurance for you and your family, a flexible vacation policy, and generous parental leave.

Compensation and Values

At Horizon3, we believe that our people are our greatest asset, and our compensation philosophy reflects this core value. We are committed to fostering an environment where all employees feel valued, respected, and rewarded for their contributions. Our compensation structure is designed to be fair, competitive, and transparent, ensuring that every team member is recognized and compensated equitably across roles, levels, and locations.

In accordance with various State’s transparency regulations, we provide the following salary range information for this position:

  • Base salary range: $211,000 - $249,000 annually. The exact salary will be determined based on the selected candidate’s location, qualifications, experience, and relevant skills.
  • Additional compensation: All full-time roles are eligible for an equity package in the form of stock options.

You Belong Here

Our commitment to diversity and inclusion means we strive to attract, develop, and retain a workforce that reflects the varied communities we serve. We believe that diverse perspectives drive innovation and strengthen our ability to create cutting-edge cybersecurity solutions. At Horizon3, every team member is valued and supported in an environment that encourages personal and professional growth.

We welcome candidates from all backgrounds and experiences, and we encourage all qualified individuals to apply. Come be a part of Horizon3, where your unique contributions are recognized, and your potential is limitless.

Other Duties

Please note this job description is not designed to cover or contain a comprehensive listing of activities, duties or responsibilities that are required of the employee. Duties, responsibilities, and activities may change at any time with or without notice.

Horizon3 is not just an equal opportunity employer - we are a community that values diversity, equity, and inclusion as fundamental principles of our culture and success. We are dedicated to fostering a workplace where everyone feels welcome and respected, regardless of race, color, religion, sex, national origin, age, disability, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, or any other legally protected status by law.

Something wrong with this job? Report it · This flag looks wrong