8020REI

Data Engineer

• Own Big Data pipelines end to end.
• Design, build, and optimize PySpark ETL/ELT pipelines on Amazon EMR and AWS Glue that process nationwide, county-partitioned property data (First American, BuildZoom permits, market comps) on daily and monthly cadences.
• Run our lakehouse.
• Operate and evolve our Bronze → Silver → Gold data lake on S3 with Apache Hudi and the AWS Glue Data Catalog, queried through Athena, including schema contracts, partitioning strategy, compaction, and performance tuning.
• Orchestrate and automate.
• Build reliable orchestration with AWS Step Functions, EventBridge, and Lambda; make reruns, backfills, and failure recovery boring and documented.
• Enforce data quality.
• Implement and extend our Data QA Audit Standard: layer contracts, write-audit-publish gating, quarantine flows, drift monitoring, and actionable Slack alerting, so bad data never reaches a client list.
• Operate production databases and APIs.
• Manage Aurora PostgreSQL and DynamoDB workloads, and run data-serving APIs (API Gateway, SQS-backed async workers), such as our Address Resolution Service, to production SLAs with dashboards and runbooks.
• Own cloud cost.
• Monitor, report, and reduce the AWS data-platform bill (EMR cluster sizing, Glue/Lambda usage, S3 lifecycle, Athena scan costs) as a first-class engineering responsibility.
• Ship infrastructure as code.
• Define infrastructure with Terraform and CloudFormation, delivered through GitHub Actions CI/CD with tests, linting, and coverage gates, we run a disciplined PR, branch-policy, and code-review culture.
• Partner with Data Science.
• Build the feature pipelines, training datasets, and serving paths behind our ML scoring and valuation models (scikit-learn/XGBoost-family stack), and co-own the handoff contracts between DS and DE.
• Document like a pro.
• Maintain runbooks, architecture docs, and data dictionaries (Confluence) so any teammate can operate what you build.

Senior Data Engineer