RunBuggy

Data Engineer

• Design, develop, and maintain scalable data pipelines and systems.
• Independently create and own new data capture/ETL’s for the entire stack and ensure data quality.
• Collaborate with data scientists, engineers, business leaders, and other stakeholders to understand data requirements and provide the necessary infrastructure.
• Create and contribute to frameworks that improve the effectiveness of logging data, triage issues, and resolution.
• Define and manage Service Level Agreements (SLA) for all data sets in allocated areas of ownership.
• Lead data engineering projects and determine the appropriate tools and libraries for each task.
• Implement data security and privacy best practices.
• Create and maintain technical documentation for data engineering processes.
• Work with cloud-based data storage and processing solutions (for example, Docker and Kubernetes).
• Build out and support a DAG orchestration cluster framework.
• Migrate workflows from batch processes to the DAG cluster via concurrent data flows.
• Data pipeline maintenance, including debugging code, monitoring, and incident response.
• Collaborate with engineering to enforce data collection and data contracts for API’s, databases, etc.
• Optimize pipelines, dashboards, frameworks, and systems to facilitate easier development of data artifacts.