Data Engineer – Mid-level
Join Creative Tech
full-time
Posted on:
Location Type: Remote
Location: Remote • 🇧🇷 Brazil
Visit company websiteJob Level
Mid-LevelSenior
Tech Stack
CloudETLPySparkSQL
About the role
- Build, test, and maintain ingestion and transformation pipelines (ETL/ELT);
- Use Databricks with a focus on PySpark and SQL;
- Ensure scalability and robustness for the BI team;
- Optimize pipeline execution;
- Perform query tuning;
- Tune clusters and Delta tables to maximize performance and reduce costs;
- Apply security and quality policies to pipelines;
- Implement governance across Bronze → Silver → Gold layers;
- Ensure quality and traceability of transformations;
- Create connections and ingest data from diverse systems;
- Ensure integrations are reliable and performant;
- Implement automated data quality rules;
- Develop gates in Databricks / Delta Live Tables (DLT);
- Ensure only qualified data reaches the consumption layer;
- Model structures in the Lakehouse;
- Document and maintain metadata repositories and models;
- Support administration of the Databricks workspace;
- Manage access and enforce information security best practices.
Requirements
- Bachelor's degree in IT or any field with a postgraduate degree in IT;
- Minimum of 4 years' experience in Software Development, Data Engineering and/or Data Administration;
- Hands-on experience with Databricks (clusters, notebooks, Delta Lake, DLT);
- Knowledge of Big Data and distributed processing;
- Strong knowledge of PySpark;
- Proficiency in SQL;
- Experience with version control tools (Git);
- Logical and physical modeling in a Lakehouse environment;
- Documentation and maintenance of metadata;
- Experience ingesting data from RDBMS, APIs, and Cloud Storage;
- Application of security rules and governance policies;
- Implementation of Data Quality Gates;
- Access management in the Databricks Workspace.
Benefits
- Infrastructure allowance;
- Flexible working hours;
- Birthday day off — with a surprise!
- Support for training and certifications;
- Access to Alura;
- Partnership with FIAP;
- Referral bonus;
- Health insurance;
- Dental plan;
- Vittude — mental health platform;
- Wellhub — for physical health;
- New Value — discount vouchers;
- SESC;
- Meal and/or food allowance.
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
ETLELTDatabricksPySparkSQLquery tuningDelta tablesautomated data quality rulesdata modelingmetadata management
Soft skills
organizational skillscommunication skillsproblem-solvingattention to detailcollaboration
Certifications
Bachelor's degree in ITPostgraduate degree in IT