Senior Data Engineer

Effectual

Senior Data Engineer specializing in data streaming technologies for Effectual. Responsible for building and maintaining high-performance data streaming architectures enabling real-time data processing and analytics.

Posted 6/16/2026full-timeRemote • 🇺🇸 United StatesSenior💰 $150,000 - $180,000 per yearWebsite

Tech Stack

Tools & technologies

ApacheAWSCloudETLKafkaMicroservicesSparkTerraform

About the role

Key responsibilities & impact

Design, build, and maintain scalable streaming data architectures using Kafka, MSK, and Kinesis
Develop real-time data pipelines that handle high-volume, high-velocity data streams
Implement event-driven architectures and microservices patterns for streaming data processing
Create and optimize data streaming topologies for complex event processing scenarios
Design fault-tolerant streaming systems with proper error handling and data recovery mechanisms
Configure, deploy, and manage Apache Kafka clusters and AWS MSK environments
Implement Kafka Connect pipelines for streaming data integration
Design optimal Kafka topic partitioning strategies and replication configurations
Monitor and optimize Kafka cluster performance, throughput, and latency
Implement Kafka security configurations including SSL/TLS, SASL, and ACLs
Manage Kafka Schema Registry for data serialization and evolution
Design and implement Amazon Kinesis Data Streams and Kinesis Data Firehose solutions
Configure Kinesis Analytics applications for real-time stream processing
Optimize Kinesis shard management and auto-scaling configurations
Implement Kinesis data retention and archival strategies
Integrate Kinesis with other AWS services for comprehensive streaming solutions
Develop real-time stream processing applications using Apache Spark Streaming, Kafka Streams, or AWS Lambda
Implement complex event processing (CEP) patterns for real-time analytics
Build streaming ETL pipelines that transform data in motion
Create real-time aggregations, windowing operations, and stateful stream processing
Optimize streaming query performance and resource utilization
Ensure seamless integration between streaming systems and data lakes, data warehouses, and operational databases
Implement data lineage and monitoring for streaming data pipelines
Create automated data quality checks and validation for streaming data
Manage data serialization formats (Avro, JSON, Protobuf) and schema evolution
Coordinate with data scientists and analysts to ensure streaming data meets analytical requirements
Implement Infrastructure as Code (IaC) for streaming data platforms using Terraform or CloudFormation
Automate deployment and management of streaming infrastructure through CI/CD pipelines
Monitor streaming system health, performance metrics, and alerting
Implement disaster recovery and high availability strategies for streaming systems
Stay current with emerging trends in streaming technologies and cloud-native solutions
Collaborate with data architects, data scientists, and application teams on streaming data requirements
Support rigorous project governance through daily progress reviews and time tracking
Provide technical leadership and mentorship to junior data engineers
Communicate complex streaming concepts to technical and non-technical stakeholders
Operate with transparency and responsiveness to support high-performing teams.

Requirements

What you’ll need

7+ years of experience in the data engineering field with significant streaming data specialization
Bachelor's degree in Computer Science, Engineering, or related STEM field
Extensive hands-on experience with Apache Kafka including cluster management, performance tuning, and ecosystem tools
Proven experience with AWS MSK and Amazon Kinesis services in production environments
Strong background in real-time data processing and stream analytics.

Benefits

Comp & perks

Medical, dental, and vision health insurances,
Short term disability, long term disability and life insurances,
401k with Company match
Paid time off (PTO) (120 hours PTO that accrue over one year)
Paid time off for major holidays (14 days per year)
These and any other employee benefit offerings are subject to management’s discretion and may change at any time.

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

Apache KafkaAWS MSKAmazon KinesisApache Spark StreamingKafka ConnectTerraformCloudFormationCI/CDdata serialization formatscomplex event processing

Soft Skills

technical leadershipmentorshipcommunicationcollaborationproject governancetransparencyresponsiveness