Senior Data Engineer - Databricks
NDEAVOUR CONSULTING · Remote · All Cities ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Senior Data Engineer
- Databricks & PySpark
- AWS Cloud Environment
- Hybrid Work Model
- Health Insurance Included
As a subject matter expert for the next-generation Databricks Lakehouse, you will design robust ingestion pipelines and set production best practices. You will establish CI/CD standards using Databricks Asset Bundles and elevate the technical capabilities of the engineering team.
The environment is hosted on AWS, though expertise in Azure or other clouds is welcome. You will tackle unstructured data sources, optimize distributed workloads, and drive architectural standards in a fast-paced rebuild setting.
What you'll do
- Lead architecture and implementation of Lakehouse ingestion pipelines using Databricks
- Establish CI/CD pipelines using Databricks Asset Bundles and orchestration tools
- Architect systems to ingest structured and unstructured data into analytical datasets
- Enforce Delta Lake principles and integrate data quality frameworks like Great Expectations
- Direct AI agents for code generation while ensuring quality and security standards
- Mentor peers and articulate technical trade-offs to stakeholders
What you'll need
- 5+ years of data engineering experience with large-scale Databricks production exposure
- Strong proficiency in AWS, Azure, or GCP cloud environments
- Hands-on experience with enterprise CI/CD for Databricks workflows
- Proven ability to process raw, unstructured data into structured Lakehouse layers
- Expert SQL optimization and deep Python proficiency
- Experience with workflow orchestrators like Airflow and Unity Catalog governance
Stack
- Databricks
- PySpark
- AWS
- Airflow
- Delta Lake
- Unity Catalog
- Great Expectations
- Databricks Asset Bundles
How you'll work
- Flexible hybrid working model
- Remote office option available
What you get
- Free parking spots provided
- Game zone and relaxation area in office
- Private health insurance with dental care plan
- Company-sponsored training budget for personal development
- Bonus for employee referrals
- Extra 5 days of holiday after 1st and 5th year
- Family member insurance option
- 100% sponsored sports cards