VOX SOLUTIONS

Data Engineer

• Event Ingestion & Streaming (Kafka KRaft)
• Build and maintain Kafka ingestion pipelines
• Define topic structures, partition strategies, retention policies, and consumer logic for multi-tenant setups
• Manage data contracts and schema evolution
• Develop idempotent ingestion services that land data into Iceberg tables
• Lakehouse Architecture (Iceberg + Nessie)
• Design and optimize Iceberg tables (partitioning, compaction, clustering, retention rules)
• Work with Nessie branches/tags to manage multi-environment (dev/test/prod) and multi-MNO deployments
• Implement Python/Spark loaders writing from Kafka → Iceberg
• Manage Iceberg compaction, metadata pruning, snapshot control, and performance tuning
• Distributed Processing & Enrichment (Spark)
• Develop Spark jobs (batch + micro-batch where needed) for: Cleaning and normalizing events, Categorizing senders, Engagement signals, Identity stitching and grouping, Audience enrichment and behavioral metrics
• Ensure Spark jobs scale efficiently across large volumes of event data
• Query Layer & Transformations (Dremio + dbt)
• Build dbt models on top of Iceberg datasets via Dremio and dbt
• Deliver telecom-specific analytical models including: Descriptive Analytics, Quality/quantity audience scoring, Campaign performance metrics, RFU relevance scoring, Cohort segmentation pipelines
• Optimise Dremio queries using reflections, column pruning, and Iceberg metadata
• Kubernetes, Helm, and CI/CD
• Maintain Helm charts for each VOX deployment (multiple clusters)
• Build CI/CD pipelines (GitHub Actions/GitLab/Argo) for: Ingestion services, Spark job deployment, Kafka topic configs, Dbt model updates, Helm releases into customer clusters
• Automate rollouts, config updates, and monitoring installation
• Observe-ability, Quality & Governance
• Implement monitoring for ingestion lag, consumer errors, Iceberg table health, Spark jobs, and Dremio performance
• Implement custom Python-based data validation checks where needed
• Handle all PII with strict tenant isolation and encryption (Vault)
• Ensure compliant and governed data flows across all deployments.
• Collaboration & Product Development
• Work with product teams to translate telecom and marketing requirements into scalable data models
• Support analysts and ML engineers with clean, enriched datasets from Iceberg
• Collaborate with DevOps on cluster performance, scaling, and stability