BLACKBIRD.AI

Data Engineer

• Design and implement scalable data platform architecture on Databricks, supporting both batch and streaming ingestion
• Build robust, fault-tolerant data ingestion pipelines that integrate with multiple third-party APIs and data providers
• Design and implement AI-powered enrichment stages within pipelines—applying ML clustering, generative AI summarization, classification, and entity extraction to transform raw data into actionable intelligence
• Build analytical systems with full-text search capabilities using Elasticsearch for rapid querying and analysis of enriched data
• Work with AI/ML researchers to implement, integrate and scaling AI processing
• Expose data platform capabilities as APIs and other interfaces for downstream consumption by applications and services
• Optimize data lake and lakehouse architecture for performance, cost-efficiency, and scalability
• Design and implement data quality frameworks, monitoring, and alerting systems
• Design efficient architectures for calling external AI APIs and managing rate limits, costs, and reliability
• Architect solutions with cost-efficiency as a first-class concern, implementing monitoring and optimization strategies for compute and storage
• Make critical build-vs-buy decisions and establish architectural standards for the data organization
• Mentor engineers and elevate the team's technical capabilities through code reviews, design discussions, and knowledge sharing

Staff Data Engineer