nDeavour Consulting

Senior Data Engineer

nDeavour Consulting

full-time

Posted on:

Location Type: Remote

Location: Remote • 🇺🇸 United States

Visit company website
✨ AI Apply
Apply

Job Level

Senior

Tech Stack

AirflowAWSCloudKafkaPySparkPythonSQLUnity

About the role

  • Design and implement robust, scalable data ingestion and transformation pipelines using Databricks, PySpark, and distributed processing. Utilize Airflow (or similar) for reliable workflow orchestration.
  • Implement Delta Lake principles, focusing on CDC and schema evolution. Establish and integrate data quality frameworks (e.g., Great Expectations) within CI/CD pipelines for data integrity.
  • Develop and optimize complex SQL and Python scripts. Integrate diverse data sources (APIs, S3 files, etc.) and handle both structured and unstructured data.
  • Support the implementation of data governance and cataloguing solutions (e.g., Unity Catalog). Proactively investigate and improve inconsistent legacy datasets.
  • Guide and manage AI agents for code generation, ensuring quality and stability. Work pragmatically and collaboratively to drive technical solutions.

Requirements

  • 5+ years of professional experience in data engineering, focused on cloud and distributed processing environments.
  • Strong experience with Databricks, PySpark, distributed processing, and Delta Lake. Deep knowledge of CDC and schema evolution.
  • Expert SQL optimization and Python skills. Hands-on experience with Airflow (or similar).
  • Familiarity with relevant AWS components. Good understanding of CI/CD for data workflows and implementing data quality frameworks.
  • Knowledge of streaming (Kafka/Kinesis) and exposure to Unity Catalog or similar governance tools.
  • Proactive problem-solver; comfortable working with complex, inconsistent data without needing explicit specs.
  • Able to work independently in a fast-paced rebuild environment.
  • Collaborative, pragmatic, thrives on rapid iteration.
  • Skilled at directing AI agents to code while maintaining high quality standards.
Benefits
  • Remote Office – Flexible hybrid form of working
  • Parking Space – We provide free parking spots
  • Fun Office Space – We offer a game zone and a relaxation area
  • Health Insurance – Additional private health insurance, including a dental care plan
  • Personal Development – Company-sponsored training budget to further develop your skills
  • Employee Referral Program – Receive a bonus for referring a friend
  • Holidays – Enjoy an extra 5 days after your 1st and 5th year
  • Social Events – We love to celebrate our success together
  • Family Insurance – Add insurance to a family member
  • Offering sports cards – 100% sponsored by the company

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard skills
data ingestiondata transformationDatabricksPySparkdistributed processingAirflowDelta LakeSQL optimizationPythondata quality frameworks
Soft skills
proactive problem-solvercollaborativepragmaticindependentcomfortable with complex dataability to drive technical solutionsthrive on rapid iteration