Custom Software Systems, Inc.

Azure Databricks Engineer

Custom Software Systems, Inc.

contract

Posted on:

Location Type: Remote

Location: Remote • Virginia • 🇺🇸 United States

Visit company website
✨ AI Apply
Apply

Job Level

Mid-LevelSenior

Tech Stack

ApacheAzureCloudETLPythonScalaSparkSQLTerraform

About the role

  • Design and build scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and Apache Spark
  • Lead the technical migration and modernization of a legacy on-premises Natural Language Processing (NLP) solution to Azure-native tools
  • Develop comprehensive real-time data processing solutions using Structured Streaming
  • Collaborate with data scientists to deploy ML models into production environments
  • Develop and maintain Terraform Infrastructure as Code (IaC) scripts for cloud platform operational builds
  • Implement advanced data security capabilities

Requirements

  • 5+ years of progressive experience as a Data Engineer
  • 3+ years of specialized, hands-on experience specifically in Azure Databricks and related Azure data ecosystem services, ideally in regulated or government environments where data security, governance, and compliance are paramount.
  • Expert-level proficiency in Apache Spark and Delta Lake
  • Advanced capabilities in Python, SQL, and Scala for developing robust data engineering solutions
  • Proven experience architecting and optimizing large-scale data pipelines handling terabytes of data.
  • Comprehensive knowledge of both batch and real-time ETL/ELT processes
  • Exceptional analytical and problem-solving abilities
  • Microsoft Certified Azure Data Engineer Associate (Preferred)
  • Databricks Certified Data Engineer Professional or Associate (Preferred)
  • BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent professional experience.
Benefits
  • Health insurance plans
  • Health Savings Account (HSA)
  • Dental
  • Vision
  • Long-term disability
  • Short-term disability
  • Basic term life insurance
  • Supplemental term life insurance for employees, spouses, and dependents
  • Simple IRA
  • Parking/Commuting expense reimbursement
  • Training/Education

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard skills
ETLELTAzure DatabricksDelta LakeApache SparkStructured StreamingTerraformPythonSQLScala
Soft skills
analytical skillsproblem-solving
Certifications
Microsoft Certified Azure Data Engineer AssociateDatabricks Certified Data Engineer ProfessionalDatabricks Certified Data Engineer Associate