Cloud and Data Engineer

MagicOrange

Cloud and Data Engineer responsible for building scalable backend solutions for data needs at a SaaS company. Collaborate on data integration, algorithm support, and optimization tasks.

Posted 7/5/2026full-timeMount Edgecombe • 🇿🇦 South AfricaMid-LevelSeniorWebsite

Core Competencies

Role fit

Core Competencies

Use this summary to align your resume positioning with the role.

Demonstrates expertise in optimizing data pipelines and ETL processes, with a strong focus on automated testing and data quality assurance within Databricks. Proficient in collaborating with cross-functional teams to deliver data-driven solutions that meet business needs while ensuring data integrity and accuracy.

Highest-signal resume keywords

Databricks ExpertiseAutomated Testing with PytestData Processing FrameworksCloud Engineering ProficiencyAI/ML Pipeline Experience

ATS Keywords

Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills

SQLPythonDatabricksApache SparkHadoopKafkaDelta LakeNoSQL DatabasesAutomated TestingData Integration

Soft Skills

Problem-Solving AbilitiesAttention to DetailExcellent Communication SkillsTime ManagementSolution Driven

Tools & Technologies

AzureAWSGoogle CloudAzure OpenAIAzure AI SearchGitDelta Live TablesGreat ExpectationsSemantic KernelLangChain

Industry Keywords

Data EngineeringSaaS EnvironmentAI Agentic WorkflowsData Quality FrameworksCI/CD Pipelines

Tech Stack

Tools & technologies

ApacheAWSAzureCloudETLHadoopKafkaNoSQLPySparkPythonSparkSQL

About the role

Key responsibilities & impact

Optimize and enhance existing data pipelines, ETL processes, and data workflows to improve performance, scalability and reliability.
Implement best practices for data processing, storage, and retrieval.
Implement automated testing within Databricks using frameworks such as pytest and nutter: write unit tests for notebooks and PySpark transformations, integration tests across pipeline stages, and data quality assertions, with results surfaced through CI/CD pipelines.
Design, implement and maintain algorithms to solve technical problems related to data processing and analytics.
Collaborate with data scientists to support and optimize machine learning algorithms and models in production.
Assist to benchmark and ensure efficient use of computational resources for data processing and algorithm execution.
Design and implement AI agentic workflows that automate multi-step data tasks: orchestrate LLM-powered agents to perform data discovery, anomaly investigation, and automated root-cause analysis within the data platform.
Apply prompt engineering and model evaluation practices when integrating large language models into data pipelines; enforce responsible-AI guardrails including output validation and human-in-the-loop review for high-impact decisions.
Design, implement and maintain solutions for data integration from various sources, ensuring data consistency and integrity.
Work on data ingestion and transformation processes to support analytics and reporting needs.
Validate integrated data using Databricks-native testing approaches: apply Delta Live Tables expectations, Great Expectations, or equivalent data quality frameworks to enforce schema, completeness, and accuracy contracts at ingestion.
Work closely with the technical product owner to understand business requirements and translate them into technical solutions.
Provide technical support and guidance to other team members regarding data-related issues.
Work closely with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
Help to document key processes and services for the purposes of sharing the load and approach to technical support related issues.
Work in conjunction with the BI Engineering to develop and maintain both product and internal dashboards and reports.
Interpret data to provide meaningful insights and recommendations to stakeholders.
Work closely with business teams to understand their and customer reporting needs and deliver tailored solutions.
Ensure data accuracy and integrity in all reports and dashboards.

Requirements

What you’ll need

3+ years of experience in cloud engineering, data engineering, or a similar role within a SaaS environment.
Demonstrable experience with Databricks: notebooks, Delta Lake, DLT pipelines, Jobs, and writing automated tests for data transformations within the platform.
Hands-on exposure to AI/ML pipelines or agentic data workflows; experience with Azure OpenAI, Azure AI Search, or equivalent services advantageous.
Strong Mathematical, Analytical, Conceptual and Problem-Solving Abilities.
Solution Driven.
Ability to find the root cause of problems and quickly determine effective solutions.
Ability to anticipate risk.
Troubleshooting, analytical and attention to details.
Ability to prioritize and manage time effectively.
Excellent Communication Skills.
Proficiency in cloud platforms such as Azure, AWS or Google Cloud.
Strong experience with data processing frameworks and tools (e.g., Databricks, Apache Spark, Hadoop, Kafka).
Strong Expertise in SQL and experience with NoSQL databases.
Familiarity with Git.
Proficiency in programming languages such as Python.
Experience building AI agentic systems: multi-agent orchestration, tool/function calling, RAG pipelines, or similar autonomous workflow patterns applied to data engineering problems.
Working knowledge of AI/LLM frameworks relevant to data engineering such as Semantic Kernel, LangChain, AutoGen, or the Azure OpenAI Service SDK; familiarity with prompt engineering and model evaluation.
Familiarity with vector databases and embedding models (e.g. Azure AI Search, Chroma, pgvector) advantageous; understanding of retrieval-augmented generation (RAG) patterns a plus.
Proficiency in automated testing within Databricks: pytest, nutter, Delta Live Tables expectations, or Great Expectations; ability to integrate test runs into Azure DevOps or equivalent CI/CD pipelines.

Benefits

Comp & perks

Strong entrepreneurial spirit.
The ability to make an impact and see the rewards of your efforts.
Ongoing training on the latest technologies to aid automation for accountants.
Be part of a high growth industry and product.
A challenging career in an innovative company.
Opportunity to influence, working in an open climate, close to decision makers at large blue-chip enterprise with the possibility to make a difference.
A competitive remuneration package, with flexible pension options.