Deeplight AI

Data Engineer

• ***Your responsibilities as a DataOps Engineer will include, but not be limited to;***
• - ***Monitoring & Alerting***
• - Monitor and act on incidents related to:
• - AWS Glue job executions
• - Current DMS and dbt pipelines
• - Kafka lag and streaming health
• - Data freshness SLA breaches
• - Data quality issues
• - Platform health alerts
• - Perform L1 triage or distribute incidents to L2/L3 teams as needed.
• - ***Incident Management***
• - Own the incident response process:
• - Initial triage and severity assessment.
• - Coordinate with development teams for resolution.
• - Create and assign JIRA tickets with full context.
• - Track incident resolution and closure.
• - Escalate high-priority or long-running incidents to management.
• - ***Root Cause Analysis & Prevention***
• - Conduct post-incident root cause analysis.
• - Maintain incident logs and post-mortem documentation.
• - Implement preventive measures for recurring issues in collaboration with dev teams.
• - ***Operational Reporting***
• - Manage the TV dashboard providing real-time status of critical flows (with color-coded indicators).
• - Deliver a daily 10–15 min operational review of previous day’s executions, open incidents, and follow-ups.
• - Share daily summary emails with the team.
• - ***Continuous Awareness***
• - Stay up-to-date on the status of all critical flows and remediation efforts.
• - Ensure proactive communication on risks and delays.

DataOps Engineer – Lakehouse Operations

• Design and enforce Medallion Architecture for the Lakehouse.
• Drive architectural decision-making through SAF/ARB architecture forums.
• Lead design for all Data Factory squads working on data initiatives.
• Define standards for Apache Iceberg tables, including partitioning strategy, schema evolution, time travel and snapshot retention, and table optimization schedules.
• Establish data modeling standards (e.g., dimensional modeling for Gold layer, SCD2, naming conventions, metadata management).
• Define lineage standards leveraging OpenMetadata.
• Coordinate with platform teams on Iceberg-related activities.
• Collaborate with Data Quality leads to implement validation rules.
• Proactively manage escalations and communicate progress with senior stakeholders.
• Provide guidance to Glue Data Engineers.
• Establish code review processes and best practices.
• Create reusable design patterns and templates for consistency.

Data Architect

• ***Your responsibilities as the AWS Glue Data Engineer will include:***
• - **Data Ingestion Development**
• - Building and implementing AWS Glue jobs for Bronze layer ingestion using defined standards and templates.
• - Implementing correct loading methods based on source requirements (CDC, full load, delta, snapshot).
• - Designing and executing historical loading mechanisms to bring legacy data into the Lakehouse.
• - **Performance Optimisation**
• - Optimising Glue job performance (DPU allocation, parallelization, partitioning) according to best practices.
• - Collaborating with platform teams to ensure tooling and optimization alignment.
• - **Migration & Automation**
• - Aggressively migrating source tables to Bronze layer, initially using manual approaches with standards/templates, later leveraging AI-enabled acceleration.
• - Ensuring jobs are version-controlled and production deployment is automated via Git and Terraform.
• - **Governance & Monitoring**
• - Implementing source system connectivity into CDP in collaboration with source system owners.
• - Ensuring jobs comply with data contracts and are properly monitored.
• - Preparing documentation and handover to operational support teams.
• - **Collaboration**
• - Working closely with Data Architect for ingestion patterns and standards.
• - Coordinating with Data Assurance Lead to apply quality checks across all jobs.
• - Partnering with platform engineers for tooling and optimisation.