Deeplight AI

Data Engineer

• ***Your responsibilities as the AWS Glue Data Engineer will include:***
• - **Data Ingestion Development**
• - Building and implementing AWS Glue jobs for Bronze layer ingestion using defined standards and templates.
• - Implementing correct loading methods based on source requirements (CDC, full load, delta, snapshot).
• - Designing and executing historical loading mechanisms to bring legacy data into the Lakehouse.
• - **Performance Optimisation**
• - Optimising Glue job performance (DPU allocation, parallelization, partitioning) according to best practices.
• - Collaborating with platform teams to ensure tooling and optimization alignment.
• - **Migration & Automation**
• - Aggressively migrating source tables to Bronze layer, initially using manual approaches with standards/templates, later leveraging AI-enabled acceleration.
• - Ensuring jobs are version-controlled and production deployment is automated via Git and Terraform.
• - **Governance & Monitoring**
• - Implementing source system connectivity into CDP in collaboration with source system owners.
• - Ensuring jobs comply with data contracts and are properly monitored.
• - Preparing documentation and handover to operational support teams.
• - **Collaboration**
• - Working closely with Data Architect for ingestion patterns and standards.
• - Coordinating with Data Assurance Lead to apply quality checks across all jobs.
• - Partnering with platform engineers for tooling and optimisation.

AWS Glue Data Engineer

Machine Learning Engineer

• Design and implement AI/ML models to solve complex business problems
• Develop data processing pipelines for model training and deployment
• Implement algorithm optimizations for performance improvement
• Create prediction engines and pattern recognition systems
• Tune models based on performance metrics and feedback
• Test AI models for accuracy, bias, and robustness
• Implement AI monitoring and logging mechanisms
• Collaborate with data engineers and full-stack developers to integrate AI solutions
• Provide technical expertise on AI/ML best practices and methodologies

AWS Glue Data Engineer

AI/ML Engineer