ANBIMA

Data Engineer – Mid-level

ANBIMA

full-time

Posted on:

Location Type: Hybrid

Location: São Paulo • Brazil

Visit company website

Explore more

✨ AI Apply
Apply

Tech Stack

AirflowAmazon RedshiftAWSEC2ETLPythonSparkSQLUnity

About the role

  • Responsible for designing, developing, and implementing end-to-end data pipelines using Databricks for ETL, streaming, and batch processing.
  • Implement data lake, data lakehouse, and data warehouse architectures on Databricks, ensuring scalability, reliability, and performance.
  • Collaborate with stakeholders and team members to translate business requirements into technical specifications and scalable solutions.
  • Implement processes and procedures to ensure data quality, governance, security, and optimization within the Databricks environment.
  • Apply deep technical knowledge to address client challenges and expand business opportunities.
  • Manage projects, ensuring milestones are met within agreed timelines and budget constraints.
  • Ensure secure, robust CI/CD automation for data pipeline delivery.
  • Participate in and support the Prices & Indices on-call rotation once a week.

Requirements

  • Experience in Data Engineering with a focus on Databricks.
  • Experience designing, building, and optimizing data solutions using Databricks, including implementing pipelines, Airflow, Databricks Workflows, Delta Lake, Unity Catalog, Spark, Python, SQL, etc.
  • Experience with cloud platforms, preferably AWS (EC2, S3, Redshift, tokens), integrated with Databricks.
  • Hands-on experience with Spark, Python, and SQL for data manipulation.
  • Experience with code versioning using GitLab.
  • Experience implementing CI/CD using GitFlow concepts.
  • Experience designing data architectures using Databricks and AWS, applying advanced concepts such as Data Lake, Data Lakehouse, Data Mesh, Data Sharing, Redshift, and integrations with data tools to ensure scalability, security, and reliability.
  • Excellent verbal and written communication skills are essential for this position.
  • Knowledge of Dataiku is a plus.
Benefits
  • Competitive salary
  • Meal allowance
  • Grocery allowance
  • Health insurance
  • Dental insurance
  • Hybrid work model
  • Life insurance
  • Private pension
  • Home office allowance
  • 6 months parental leave for all ANBIMA employees
  • Funeral assistance
  • Partnerships, birthday day off, and much more...
Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools
Data EngineeringDatabricksETLdata pipelinesAirflowDatabricks WorkflowsDelta LakeSparkPythonSQL
Soft Skills
communicationcollaborationproject managementdata qualitygovernancesecurityoptimization