Data Architect – Databricks
Allata
AirflowAWSAzureCloudETLGrafanaJenkinsMS SQL ServerOraclePostgresPrometheusPySpark+4 more
Salary
💰 $130,000 - $160,000 per year
Job Level
Mid-LevelSenior
Tech Stack
AirflowAWSCloudETLPythonReactSQLTerraform
About the role
- Build and optimize data models in dbt Core to create reliable, efficient, and accessible data for downstream reporting and analysis, with a strong understanding of end-user needs.
- Design, develop, and maintain scalable data ingestion and orchestration using Meltano, Snowpipe, Airflow and other tools.
- Manage and automate data infrastructure in AWS using Terraform.
- Collaborate with Data Analysts and Software Engineers to clarify data requirements and translate them into effective data engineering solutions.
- Proactively identify and implement improvements in data orchestration, cost/performance management, and security within Snowflake.
- Develop new data ingestion pipelines from various source systems into Snowflake, including full-stack development for brand-new pipelines from ingestion to data modeling of core user-facing tables.
- Implement efficient testing within dbt to detect system changes and ensure data quality, contributing to the operational health of the data platform.
Requirements
- 3+ years experience as a Data Engineer with a strong focus on data pipeline development and data warehousing, consistently delivering high-quality work on a timely basis.
- Strong hands-on experience with data modeling, knowledgeable about general design patterns and architectural approaches.
- Hands-on experience with cloud data warehouses.
- Strong Python and SQL skills and experience with data manipulation and analysis, capable of quickly absorbing and synthesizing complex information.
- Experience with data ingestion tools and ETL/ELT processes.
- Experience with Airflow.
- A proactive mindset to keep an eye out for areas to improve our data infrastructure.
- Ability to independently define projects, clarify requirements while considering solutions with the help of mentorship for complex projects.
- Excellent problem-solving skills and attention to detail, with a high-level understanding of how downstream users leverage data.
