Senior Data Engineer
ThetaRay
AirflowApacheAzureDockerHadoopHDFSJenkinsKubernetesLinuxMicroservicesPySparkScala+2 more
Job Level
Mid-LevelSenior
Tech Stack
AirflowAmazon RedshiftAzureBigQueryCloudDockerETLGoogle Cloud PlatformJenkinsKafkaKubernetesNeo4jPrometheusPySparkPythonRabbitMQSparkSQLTerraform
About the role
- Provide the direction of our data architecture. Determine the right tools for the right jobs. We collaborate on the requirements and then you call the shots on what gets built. Manage end-to-end execution of high-performance, large-scale data-driven projects, including design, implementation, and ongoing maintenance. Optimize and monitor the team-related cloud costs. Design and construct monitoring tools to ensure the efficiency and reliability of data processes. Implement CI/CD for Data Workflows
Requirements
- 3+ Years of Experience in data engineering and big data at large scales. - Must Extensive experience with modern data stack - Must: Snowflake, Delta Lake, Iceberg, BigQuery, Redshift Kafka, RabbitMQ, or similar for real-time data processing. Pyspark, Databricks Strong software development background with Python/OOP and hands-on experience in building large-scale data pipelines. - Must Hands-on experience with Docker and Kubernetes. - Must Expertise in ETL development, data modeling, and data warehousing best practices. Knowledge of monitoring & observability (Datadog, Prometheus, ELK, etc) Experience with infrastructure as code, deployment automation, and CI/CD. Practices using tools such as Helm, ArgoCD, Terraform, GitHub Actions, and Jenkins.
Similar jobs on JobTailor
Senior Data Engineer
ControlUp
AWSAzureCloudGoogle Cloud PlatformJavaJavaScriptKafkaMicroservicesMySQLNode.jsPostgresRedis+2 more
Data Engineer, Team Lead
Deloitte
AirflowBigQueryCloudETLJavaPythonScalaSparkSQL
Staff Data Architect
Cellebrite
Amazon RedshiftApacheAWSCloudETLKafkaPythonSparkSQL
Senior Data Engineer
Razor Labs
AirflowApacheBigQueryCloudETLGoogle Cloud PlatformNumpyPandasPythonSparkSQL