Data Engineer jobs

Data Lead

PhotonUnited StatesFull-timeHybrid10+ years experience

$46k/yr listed by the employer

Looks real. 1 thing to check before you apply.

On Photon’s own careers site (Oracle Recruiting), checked open today.

  • To check: Posted by the employer 7 months agoLong-open jobs are sometimes already filled. Ask when they plan to hire.
  • Checked: On the company's own careers siteOracle Recruiting board
  • Checked: Checked open todayOracle Recruiting board
  • Checked: Pay listedJob posting
  • Checked: No reposts foundOracle Recruiting board
  • Trust score 80 / 100 · Looks good
See all 6 checksShow fewer
How we check

What you’ll do

  • Design and implement scalable ETL/ELT pipelines for structured and unstructured data used by LLMs.
  • Build and optimize vector databases, retrieval architectures, chunking, and embedding strategies.
  • Develop data-cleaning workflows and enrich data with metadata for agent retrieval and reasoning.
  • Build low-latency data streams and create gold datasets and versioned snapshots to benchmark agent performance.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 10+ years of experience in Data Engineering
  • At least 2 years focused on data for LLMs or AI/ML applications
  • Strong experience with modern data stack tools
  • Hands-on experience with at least one major Vector Database
  • Experience with LlamaIndex or LangChain for data ingestion (preferred)

Skills

  • Deep expertise in Python, including Pandas, Pydantic, and FastAPI
  • Knowledge of similarity search algorithms, including HNSW and Cosine Similarity
  • Familiarity with hybrid search techniques combining semantic and keyword search
  • Proficiency managing data workloads on AWS, Azure, or GCP
  • Knowledge of Graph Databases such as Neo4j (preferred)
  • Familiarity with Data-Centric AI principles (preferred)

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Medical benefits for full-time employees
Dental & vision
Dental and vision benefits for full-time employees
Retirement plan
401(k) retirement plan for full-time employees
Paid time off
Paid time off and paid holidays for full-time employees
Bonus
Variable pay and incentives for full-time employees

Full job description

From Photon’s careers site

We are seeking a Lead Data Engineer to build and scale the data infrastructure powering our Agentic AI products. You will be responsible for the "Ingestion-to-Insight" pipeline that allows autonomous agents to access, search, and reason over vast amounts of proprietary and public data.

Your role is critical: you will design the RAG (Retrieval-Augmented Generation) architectures and data pipelines that ensure our agents have the right context at the right time to make accurate decisions.

Key Responsibilities

  • AI-Ready Data Pipelines: Design and implement scalable ETL/ELT pipelines that process both structured (SQL, logs) and unstructured (PDFs, emails, docs) data specifically for LLM consumption.
Show full descriptionShow less
  • Vector Database Management: Architect and optimize Vector Databases (e.g., Pinecone, Weaviate, Milvus, or Qdrant) to ensure high-speed, relevant similarity searches for agentic retrieval.
  • Chunking & Embedding Strategies: Collaborate with AI Engineers to optimize data chunking strategies and embedding models to improve the "recall" and "precision" of the agent's knowledge retrieval.
  • Data Quality for AI: Develop automated "Data Cleaning" workflows to remove noise, PII (Personally Identifiable Information), and toxicity from training/context datasets.
  • Metadata Engineering: Enrich raw data with advanced metadata tagging to help agents filter and prioritize information during multi-step reasoning tasks.
  • Real-time Data Streaming: Build low-latency data streams (using Kafka or Flink) to provide agents with "fresh" data, enabling them to act on real-time market or operational changes.
  • Evaluation Frameworks: Construct "Gold Datasets" and versioned data snapshots to help the team benchmark agent performance over time.

Required Skills & Qualifications

  • Experience: 10+ years in Data Engineering, with at least 2 years focusing on data for LLMs or AI/ML applications.
  • Python Mastery: Deep expertise in Python (Pandas, Pydantic, FastAPI) for data manipulation and API integration.
  • Data Tooling: Strong experience with modern data stack tools (e.g., dbt, Airflow, Dagster, Snowflake, or Databricks).
  • Vector Expertise: Hands-on experience with at least one major Vector Database and knowledge of similarity search algorithms (HNSW, Cosine Similarity).
  • Search Knowledge: Familiarity with hybrid search techniques (combining semantic search with traditional keyword search like Elasticsearch/BM25).
  • Cloud Infrastructure: Proficiency in managing data workloads on AWS, Azure, or GCP.

Preferred Qualifications

  • Experience with LlamaIndex or LangChain for data ingestion.
  • Knowledge of Graph Databases (e.g., Neo4j) to help agents understand complex relationships between data points.
  • Familiarity with "Data-Centric AI" principles—prioritizing data quality over model size.

Compensation, Benefits and Duration

Minimum Compensation: USD 46,000

Maximum Compensation: USD 162,000

Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.

Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.

This position is not available for independent contractors

No applications will be considered if received more than 120 days after the date of this post

Something wrong with this job? Report it · This flag looks wrong

Similar jobs

Lead Data Engineer

AIR Communities · Denver, CO

$135k–$160k/yrFull-timeOn-site

StrongChecked open todayPosted yesterday

Medicaid Data Lead

Abt Global · Rockville, MD +3 places

$110k–$155k/yr

StrongChecked open todayPosted 8 days ago

Senior Data Engineer

Vericast · Remote · US only

$130k–$150k/yrFull-time

StrongChecked open todayPosted today

Senior Data Engineer

AIR Communities · Denver, CO

$125k–$145k/yrFull-timeOn-site

StrongChecked open todayPosted yesterday

See more like this