Senior Manager, Data Engineering
Upstart
AirflowCloudETLKafkaPythonSparkSQL
Data Engineer
CACI International Inc
full-time
Posted on:
Location Type: Remote
Location: Remote • 🇺🇸 United States
Visit company websiteSalary
💰 $75,200 - $158,100 per year
Job Level
Mid-LevelSenior
Tech Stack
AWSCloudElasticSearchETLNoSQLPython
About the role
- Design and implement scalable data pipelines for dark web and open-source intelligence data
- Architect and maintain robust ETL/ELT processes for multi-source data integration
- Develop and optimize data models for complex intelligence datasets
- Implement data quality frameworks ensuring integrity of intelligence data
- Design, implement, and maintain Elasticsearch clusters optimized for intelligence data
- Implement advanced search capabilities and query optimization across terabytes of dark web data
- Create and maintain Kibana dashboards and visualizations for intelligence analysts
- Build and maintain ML data pipelines supporting threat detection and predictive analytics
- Collaborate with data scientists to productionize machine learning models
- Identify and resolve performance bottlenecks in large-scale data processing systems
- Design fault-tolerant, highly available systems for mission-critical operations
- Implement monitoring, alerting, and observability solutions for data infrastructure
- Collaborate with intelligence analysts to translate requirements into technical solutions
- Develop metrics and KPIs to measure data quality and system performance
Requirements
- 5+ years of experience in data engineering with focus on large-scale data processing
- Expert-level proficiency in Elasticsearch including cluster architecture and optimization
- Strong programming skills in Python
- Database expertise across relational and NoSQL technologies
- Cloud platform experience (AWS) with infrastructure-as-code tools
- Experience with containerization and orchestration technologies
- Strategic thinking capability with strong analytical and troubleshooting skills
- Experience with data modeling for analytical workloads
- Understanding of statistical concepts and machine learning principles
- Self-motivated and autonomous work style
- Excellent communication skills with both technical and non-technical stakeholders
- Bachelor's degree in Computer Science, Data Engineering, or related field
Benefits
- healthcare
- wellness
- financial
- retirement
- family support
- continuing education
- time off benefits
Applicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard skills
data engineeringETLELTdata modelingElasticsearchPythonNoSQLcloud computingmachine learningdata quality frameworks
Soft skills
strategic thinkinganalytical skillstroubleshooting skillsself-motivatedautonomous work stylecommunication skills
