Data Engineer jobs

Lead Data Engineer

Black Canyon ConsultingRemote · US onlyPosted 1 month ago

Pay not listed

Key facts

Experience
6+ years
Degree
Bachelor's degree

Looks real. 1 thing to check before you apply.

On Black Canyon Consulting’s own careers site (Greenhouse), checked open today.

  • To check: No pay range listed, though one is likely required in MDAsk for the pay range before you interview. The law here likely requires one.
  • Checked: On the company's own careers siteGreenhouse board
  • Checked: Checked open todayGreenhouse board
  • Checked: No reposts foundGreenhouse board
  • Trust score 89 / 100 · Looks good
See all 5 checksShow fewer
How we check

What you’ll do

  • Design and implement a DuckDB and Apache Iceberg analytical data platform on AWS, including table organization and optimization strategies.
  • Establish reusable layered data-engineering patterns and develop analytical datasets, certified data products, and business metrics.
  • Lead data migration and harmonization across source systems, implement canonical mappings, and build incremental loading and reconstruction patterns.
  • Implement agentic and AI-enabled workflows, automated reconciliation, data-quality monitoring, testing, and operational observability.

Summarised from the posting by AI and can be wrong. The full description is the source.

Requirements

Experience

  • 6+ years of relevant data engineering or data platform experience, including experience leading or mentoring other engineers
  • Experience building cloud-based analytical platforms using object storage and columnar formats such as Parquet
  • Experience implementing bronze/silver/gold or comparable layered data architectures and transforming heterogeneous enterprise datasets
  • Experience with AWS data services, particularly Amazon S3 and AWS Glue
  • Experience designing reusable, metadata-driven ingestion and transformation frameworks
  • Experience implementing automated data quality, reconciliation, monitoring, and operational controls
  • Experience with agentic or AI-enabled data engineering solutions for data mapping, pipeline generation, validation, testing, or troubleshooting
  • Hands-on experience with agentic frameworks such as LangChain, LangGraph, or similar frameworks for building tool-using or multi-step AI workflows
  • Experience with Kafka or Amazon MSK and CDC architectures (preferred)
  • Experience with GeoParquet, PostGIS, or geospatial datasets (preferred)
  • Experience with OpenSearch, document stores, or vector search technologies (preferred)
  • Experience integrating enterprise SaaS platforms, APIs, databases, and file-based sources (preferred)
  • Experience working in the U.S. Federal Government space (preferred)
  • Experience with permitting, regulatory, land management, environmental, energy, or natural-resource datasets (preferred)

Education

  • Bachelor's degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related field

Skills

  • Strong hands-on experience with DuckDB and Apache Iceberg
  • Advanced SQL and strong Python development experience
  • Understanding of partitioning, schema evolution, snapshots, compaction, incremental processing, and analytical query optimization
  • Strong troubleshooting and performance tuning skills
  • Technical leadership skills

Your match

Sign in and add your resume to see which of this job's requirements it shows.

Sign in to check your match

Saved to your account, encrypted. The AI that reads it doesn't keep it or train on it.

Benefits

Health insurance
Medical coverage
Dental & vision
Dental and vision coverage
Retirement plan
401(k) plan with employer contribution
Paid time off
Paid holidays and vacation
Learning & tuition
Tuition reimbursement

Full job description

From Black Canyon Consulting’s careers site

Overview

We are seeking a Lead Data Engineer to design and implement the core analytical data platform for a federal permitting modernization initiative.

The Lead Data Engineer will establish a modern lakehouse environment using DuckDB, Apache Iceberg, Amazon S3, and AWS Glue Catalog and will lead the engineering patterns used to ingest, harmonize, and serve data across multiple source systems. The role will also help implement metadata-driven and agentic approaches to data engineering, including source analysis, mapping, pipeline generation, testing, reconciliation, and quality assurance.

Show full descriptionShow less

This is a hands-on technical leadership role with responsibility for guiding and reviewing the work of other Data Engineers.

This position is remote.

Duties & Responsibilities

  • Design and implement the core DuckDB / Apache Iceberg data platform on AWS.
  • Define physical data organization, schema, partitioning, file layout, snapshot, compaction, and optimization strategies for Iceberg tables.
  • Integrate DuckDB with Amazon S3, Apache Iceberg, and AWS Glue Catalog.
  • Establish bronze, silver, and gold data-layer conventions and reusable engineering patterns.
  • Lead and mentor Data Engineers, providing technical direction, code reviews, and engineering guidance.
  • Build and maintain analytical datasets supporting application services, reporting, analytics, and data science.
  • Develop reusable patterns for incremental loads, CDC, schema evolution, late-arriving data, backfills, and historical reconstruction.
  • Work with the Data Architect to implement the common canonical data model and source-to-canonical mappings.
  • Lead migration and harmonization of datasets from multiple source systems.
  • Implement reconciliation processes that demonstrate completeness and accuracy between source and target systems.
  • Establish automated data-quality monitoring, operational observability, and testing.
  • Implement agentic and AI-enabled data engineering patterns supporting activities such as source analysis, mapping, pipeline generation, troubleshooting, reconciliation, and quality assurance.
  • Support integration of structured data with associated document and geospatial datasets.
  • Lead development of certified data products, analytical datasets, and business metrics.
  • Optimize DuckDB queries and Iceberg datasets for performance and efficiency.
  • Develop engineering documentation, operational procedures, and technical handoff materials.

Educational Requirements

  • Bachelor's degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related field.

Requirements

  • 6+ years of relevant data engineering or data platform experience, including experience leading or mentoring other engineers.
  • Strong hands-on experience with DuckDB and Apache Iceberg.
  • Advanced SQL and strong Python development experience.
  • Experience building cloud-based analytical platforms using object storage and columnar formats such as Parquet.
  • Strong understanding of partitioning, schema evolution, snapshots, compaction, incremental processing, and analytical query optimization.
  • Experience implementing bronze/silver/gold or comparable layered data architectures and transforming heterogeneous enterprise datasets.
  • Experience with AWS data services, particularly Amazon S3 and AWS Glue.
  • Experience designing reusable, metadata-driven ingestion and transformation frameworks.
  • Experience implementing automated data quality, reconciliation, monitoring, and operational controls.
  • Experience with agentic or AI-enabled data engineering solutions, including the use of agents or AI-assisted workflows for activities such as data mapping, pipeline generation, validation, testing, or troubleshooting.
  • Hands-on experience with agentic frameworks such as LangChain, LangGraph, or similar frameworks for building tool-using or multi-step AI workflows.
  • Strong troubleshooting, performance tuning, and technical leadership skills.

Bonus Skills

  • Experience with Kafka or Amazon MSK and CDC architectures.
  • Experience with GeoParquet, PostGIS, or geospatial datasets.
  • Experience with OpenSearch, document stores, or vector search technologies.
  • Experience integrating enterprise SaaS platforms, APIs, databases, and file-based sources.
  • Experience working in the U.S. Federal Government space.

Experience with permitting, regulatory, land management, environmental, energy, or natural-resource datasets.

Benefits and Salary

We attract the best people in the business with our competitive benefits package that includes medical, dental and vision coverage, 401k plan with employer contribution, paid holidays, vacation, and tuition reimbursement.

If you enjoy being a part of a high performing, professional service and technology focused organization, please apply today!

Something wrong with this job? Report it · This flag looks wrong

Similar remote jobs

Lead Data Engineer

Oddball · Remote · location not stated

$110k–$145k/yr

StrongChecked open todayPosted 15 days ago

Senior Data Engineer

Vericast · Remote · US only

$130k–$150k/yrFull-time

StrongChecked open todayPosted yesterday

Data Engineer

Mindex · Remote · US only

$90k–$140k/yrFull-time

StrongChecked open todayPosted 8 days ago

Data Integration Engineer

D.A. Davidson · Remote · US only

$74k–$106k/yrFull-time

StrongChecked open todayPosted 10 days ago

Lead Analytics Engineer

The Knot Worldwide · Remote · US only

$135k–$231k/yr

StrongChecked open todayPosted 11 days ago

Principal Data Engineer

Constant Contact · Remote · US, Canada

$138k–$172k/yrFull-time

StrongChecked open todayPosted 11 days ago

See more like this