Design, build, and maintain data ingestion pipelines that consolidate customer profile data for a real-time AI-driven decisioning application
Develop and optimize ETL processes using Azure Databricks and Azure Data Factory to source, transform, and load data from multiple upstream systems and enterprise data lakes
Ensure data quality, consistency, and reliability across batch and near real-time data flows
Manage deployment and version control of pipelines using GitLab and collaborate on code development via VSCode with GitHub Co-pilot
Monitor, troubleshoot, and optimize scheduled jobs in Control-M
Support ingestion into Postgres (for decisioning reference tables) and Cassandra (for near real-time attributes)
Work closely with other engineers, analysts, and application teams to ensure timely delivery of accurate customer data
Requirements
Bachelor's Degree Computer Science or Engineering (Required)
2-4+ years technical engineering experience (Required)
Strong SQL skills with proven experience in relational database systems (preferably PostgreSQL)
Hands-on experience with Azure Databricks (PySpark, SparkSQL, Python) for building and maintaining data pipelines
Proficiency with Azure Data Factory for pipeline orchestration and data integration
Experience with batch ETL processes at scale, including data cleansing, transformation, and performance optimization
Familiarity with Cassandra or other NoSQL databases for handling near real-time data
Version control and CI/CD using GitLab (or similar tools)
Job scheduling and monitoring experience (Control-M or equivalent)
Proficient in Python for ETL scripting and automation
Experience working in agile teams using Jira for sprint planning, backlog tracking, and defect management
Ability to work independently while maintaining clear communication and collaboration with stakeholders
Benefits
Medical, dental and vision insurance
Flexible spending account
401(k)
Employee stock grants
Employee stock purchase plan
Paid time off
Up to 12 paid holidays
Paid parental and family leave
Family building benefits
Back-up care
Enhanced family support
Childcare subsidy
Tuition assistance
College coaching
Short- and long-term disability
Voluntary AD&D coverage
Voluntary accident coverage
Voluntary life insurance
Voluntary disability insurance
Voluntary long-term care insurance
Mobile service & home internet discounts
Pet insurance
Access to commuter and transit programs
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
SQLAzure DatabricksPySparkSparkSQLPythonAzure Data FactoryETL processesCassandraPostgreSQLdata cleansing