PSA Engineer
Zone IT Solutions
JavaLinuxOracleShell ScriptingSQL
Azure Databricks Engineer
Custom Software Systems, Inc.
contract
Posted on:
Location Type: Remote
Location: Remote • Virginia • 🇺🇸 United States
Visit company websiteJob Level
Mid-LevelSenior
Tech Stack
ApacheAzureCloudETLPythonScalaSparkSQLTerraform
About the role
- Design and build scalable ETL/ELT pipelines using Azure Databricks, Delta Lake, and Apache Spark
- Lead the technical migration and modernization of a legacy on-premises Natural Language Processing (NLP) solution to Azure-native tools
- Develop comprehensive real-time data processing solutions using Structured Streaming
- Collaborate with data scientists to deploy ML models into production environments
- Develop and maintain Terraform Infrastructure as Code (IaC) scripts for cloud platform operational builds
- Implement advanced data security capabilities
Requirements
- 5+ years of progressive experience as a Data Engineer
- 3+ years of specialized, hands-on experience specifically in Azure Databricks and related Azure data ecosystem services, ideally in regulated or government environments where data security, governance, and compliance are paramount.
- Expert-level proficiency in Apache Spark and Delta Lake
- Advanced capabilities in Python, SQL, and Scala for developing robust data engineering solutions
- Proven experience architecting and optimizing large-scale data pipelines handling terabytes of data.
- Comprehensive knowledge of both batch and real-time ETL/ELT processes
- Exceptional analytical and problem-solving abilities
- Microsoft Certified Azure Data Engineer Associate (Preferred)
- Databricks Certified Data Engineer Professional or Associate (Preferred)
- BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent professional experience.
Benefits
- Health insurance plans
- Health Savings Account (HSA)
- Dental
- Vision
- Long-term disability
- Short-term disability
- Basic term life insurance
- Supplemental term life insurance for employees, spouses, and dependents
- Simple IRA
- Parking/Commuting expense reimbursement
- Training/Education
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
ETLELTAzure DatabricksDelta LakeApache SparkStructured StreamingTerraformPythonSQLScala
Soft skills
analytical skillsproblem-solving
Certifications
Microsoft Certified Azure Data Engineer AssociateDatabricks Certified Data Engineer ProfessionalDatabricks Certified Data Engineer Associate
