Imagine Believe Realize, LLC.

Data Engineer

• Design and implement secure, next-generation data engineering solutions supporting critical missions
• Collaborate with technical leaders, business stakeholders, and multidisciplinary teams
• Support implementation of the DON MAHRS program Data Strategy and High-Level Total Force Data Hub implementation plan
• Develop, implement, integrate, and sustain data architectures, data hubs, data lakes, and data warehouses
• Design and develop data models, data structures, and data acquisition processes
• Develop engineering and implementation plans for data hubs, acquisition, modeling, integration, and related activities
• Research and evaluate data sources to identify authoritative and reliable sources for data hub development
• Plan, create, and maintain data architectures aligned with business requirements
• Design, develop, maintain, and optimize ETL/ELT pipelines and transformation processes
• Develop and maintain batch and streaming pipelines using Spark, Python, Databricks, Palantir Foundry, Kafka, and related tools
• Develop data acquisition processes to ingest, transform, validate, and integrate enterprise data
• Implement incremental loading, late-arriving data, processing-window, data-freshness, and lifecycle approaches
• Automate manual data processes and improve workflow efficiency, reliability, and scalability
• Develop, maintain, and optimize solutions within Databricks and enterprise data lakes
• Configure, monitor, manage, and document Databricks clusters according to DON policies and security requirements
• Develop and maintain Spark solutions using PySpark, Spark SQL, DataFrames, and Data Sets
• Implement and maintain Delta Lake and Delta Live Tables solutions
• Optimize data processing, storage, and query performance
• Support Palantir Foundry ontologies, ETL/ELT pipelines, applications, and data-driven user interfaces
• Integrate Palantir Foundry with enterprise data engineering and analytics environments
• Develop and maintain streaming solutions using Kafka, Kafka Streams, and ksqlDB
• Configure and manage Kafka topics and Schema Registry
• Develop Python data-processing applications and AWS Lambda functions
• Support processing with AWS S3, Kinesis, Lambda, and DynamoDB
• Monitor, troubleshoot, and optimize cloud-based processing solutions
• Develop data quality controls, validation processes, and data quality gates
• Develop and execute data-driven and unit testing for Spark, Python, and other processing solutions
• Establish data lifecycle policies covering retention, backup, recovery, and data management
• Monitor and troubleshoot pipelines and processing jobs
• Improve performance, reliability, scalability, and maintainability

Senior Data Engineer