FREE ACCESS
5,000–10,000 jobs/day
See all jobs on JobTailor
Search thousands of fresh jobs every day.
Discover
- Fresh listings
- Fast filters
- No subscription required
Create a free account and start exploring right away.
Data Quality Engineer
Select Minds LLCData Quality Engineer responsible for validating data pipelines and ensuring data correctness. Focuses on building scalable data validation frameworks in AWS, Databricks, Kafka, and Python.
Tech Stack
Tools & technologiesAmazon RedshiftApacheAWSETLGrafanaKafkaPrometheusPySparkPythonScalaSparkSQL
About the role
Key responsibilities & impact- Validate data pipelines for accuracy, completeness, consistency, and timeliness
- Build SQL-based validations for business rules and transformations
- Implement reconciliation between source and downstream systems
- Ensure data lineage and traceability
- Test pipelines built on AWS (Glue, Lambda, EMR, Step Functions)
- Validate transformations using SQL and Python
- Test ingestion, transformation, aggregation, and serving layers
- Handle backfills, reprocessing, and historical data loads
- Validate Spark pipelines (PySpark/Scala) on Databricks Streaming (Kafka)
- Validate data integrity, ordering, and delivery guarantees
- Test producer and consumer logic and serialization formats (Avro, JSON, Protobuf)
- Validate topics, partitions, offsets, retention, and schema evolution
- Simulate late events, duplicates, and failure scenarios
- Build Python-based data testing frameworks
- Develop reusable validation utilities and synthetic datasets
- Integrate data tests into CI/CD pipelines
- Enable automated alerts for data quality issues
- Validate throughput, latency, and concurrency at scale
- Test retry logic, idempotency, and recovery mechanisms
- Perform regression, soak, and failover testing
- Validate logs, metrics, and alerts using tools such as CloudWatch, Prometheus, and Grafana
- Define and monitor data SLAs and SLOs
- Support incident response, root cause analysis, and postmortems
Requirements
What you’ll need- 7+ years of total experience in QA, SDET, or Data Quality Engineering
- Minimum 4–6 years of hands-on experience working with data platforms, data pipelines, or data engineering ecosystems
- 3+ years of hands-on experience with Databricks and Apache Spark
- Strong SQL skills for data validation, reconciliation, and complex analysis
- Proficiency in Python for automation and data validation
- Experience testing ETL/ELT pipelines (batch and streaming)
- Hands-on experience with Kafka or similar streaming platforms
- Strong understanding of AWS data services (S3, Glue, Lambda, Redshift, Athena)
- Experience working with large-scale distributed data systems
- Strong debugging, analytical, and problem-solving skills
Benefits
Comp & perks- Competitive salary
- Opportunity for advancement
ATS Keywords
✓ Tailor your resumeApplicant Tracking System Keywords
Tip: use these terms in your resume and cover letter to boost ATS matches.
Hard Skills & Tools
SQLPythonDatabricksApache SparkETLELTKafkaAWSData ValidationData Quality Engineering
Soft Skills
debugginganalytical skillsproblem-solving