Sparibis

Data Engineer

• Provide data engineering expertise in developing, implementing, integrating, and sustaining data architectures, data hubs, data lakes, and data warehouse solutions
• Support design and development of data models, data structures, and data acquisition processes for the HC/HR Data Domain
• Develop engineering and implementation plans for data hubs, acquisition, modeling, integration, and related data engineering activities
• Research and evaluate enterprise data sources for authoritative and reliable data hub development
• Plan, create, and maintain data architectures aligned with business requirements
• Design, develop, maintain, and optimize ETL/ELT pipelines and data transformation processes
• Develop and maintain batch and streaming pipelines using Spark, Python, Databricks, Palantir Foundry, Kafka, and related tools
• Develop data acquisition processes to ingest, transform, validate, and integrate diverse enterprise data
• Implement incremental loading, late-arriving data, processing-window, data-freshness, and lifecycle strategies
• Automate manual processes and improve workflow efficiency, reliability, and scalability
• Develop, maintain, and optimize Databricks and enterprise data-lake solutions
• Configure, monitor, manage, and document Databricks clusters according to DON policies, standards, and security requirements
• Develop and maintain Spark-based processing solutions using PySpark, Spark SQL, Data Frames, and Data Sets
• Implement and maintain Delta Lake and Delta Live Tables solutions
• Optimize data processing, storage, and query performance
• Support Palantir Foundry ontologies, ETL/ELT pipelines, applications, user interfaces, and enterprise integrations
• Develop and maintain Kafka streaming solutions, topics, Schema Registry, Kafka Streams, and ksqlDB components
• Develop Python data-processing applications and AWS Lambda functions
• Support data integration and processing with AWS S3, Kinesis, Lambda, and DynamoDB
• Monitor, troubleshoot, and optimize cloud-based data-processing solutions
• Develop data-quality controls, validation processes, and quality gates
• Develop and execute data-driven and unit testing for Spark, Python, and other processing solutions
• Establish data lifecycle policies covering retention, backup, recovery, and data management
• Monitor and troubleshoot pipelines and processing jobs, resolving quality, performance, and integration issues
• Improve data-processing performance, reliability, scalability, and maintainability