Tech Stack
AirflowApacheAWSElasticSearchKubernetesMySQLPostgresPython
About the role
- Design, build, and maintain custom OpenMetadata connectors for metadata ingestion.
- Extend and automate data lineage capture from multiple systems and repositories.
- Debug, patch, and enhance backend features to improve stability and scalability.
- Manage containerized deployments (Kubernetes/ECS) and implement monitoring solutions.
- Optimize storage and retrieval using RDS (PostgreSQL/MySQL) and OpenSearch/Elasticsearch.
- Orchestrate metadata workflows with Apache Airflow and integrate with existing pipelines.
- Implement and maintain RBAC with Okta, supporting clear role separation.
- Oversee upgrade management, backup strategies, and high-availability configurations.
Requirements
- 5+ years in data engineering or platform engineering roles, with hands-on OpenMetadata experience.
- Proficiency in ECS (AWS), Airflow, RDS, and OpenSearch/Elasticsearch.
- Strong Python development skills for API integrations and connector development.
- Proven experience implementing RBAC and identity management integrations (e.g., Okta).
- Skilled in container orchestration, infrastructure automation, and performance optimization.
- Ability to merge and manage open-source updates into customized codebases.
- Strong problem-solving skills and a track record of delivering production-grade, scalable solutions.