Senior Data Engineer, PySpark, SQL
CI&T
full-time
Posted on:
Location Type: Hybrid
Location: Brazil
Visit company websiteExplore more
Job Level
About the role
- Develop and maintain data pipelines (ETL/ELT) in an Azure environment, using Databricks for processing and integration.
- Implement CI/CD practices for automated deployments and code versioning in GitHub.
- Design and optimize data models to ensure scalability, performance and governance.
- Support cloud migration projects, ensuring data security, quality and integrity.
- Collaborate with business and data science teams to deploy AI models in cloud environments.
- Ensure data quality and consistency by applying data profiling and monitoring techniques.
- Document processes, architectures and data engineering best practices.
Requirements
- Knowledge of PySpark and SQL
- Basic knowledge of Power BI
- Strong experience with Azure Data Services and Databricks
- Proficiency in Python and SQL for data manipulation and transformation
- Knowledge of CI/CD and version control with GitHub
- Experience in data modeling (conceptual, logical and physical)
- Experience with cloud migration projects
- Experience with ETL/ELT and integrating large volumes of data
- Experience deploying AI models and using tools such as Azure ML
- Nice to have / Differentials:
- Knowledge of DB2
- Experience with AI projects and automating pipelines
- Knowledge of data-oriented architectures and governance best practices
- Participation in MDM (Master Data Management) initiatives
- Familiarity with Databricks MLflow for model management
- Experience working in Data Lake and Data Warehouse environments
Benefits
- Health and dental insurance
- Meal and food allowance
- Childcare assistance
- Extended parental leave
- Partnerships with gyms and health & wellness professionals via Wellhub (Gympass) / TotalPass
- Profit Sharing (PLR)
- Life insurance
- Continuous learning platform (CI&T University)
- Discount club
- Free online platform dedicated to promoting physical and mental health and wellbeing
- Pregnancy and responsible parenthood course
- Partnerships with online course platforms
- Language learning platform
- And many others
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
PySparkSQLPythonETLELTdata modelingCI/CDdata profilingdata integrationdata transformation