Data Engineering Manager, Big Data

Trellix

full-time

Posted on: 9/27/2025

Location Type: Hybrid

Location: Bangalore • 🇮🇳 India

✨ AI Apply

SeniorLead

ApacheAWSBigQueryCassandraCloudDistributed SystemsDockerETLHBaseJavaKafkaKubernetesMySQLNoSQLPostgresScalaSparkTerraform

About the role

Lead, mentor, and grow a team of talented data engineers
Provide technical guidance and conduct code reviews to ensure best practices
Architect and develop scalable and robust data pipelines using big data frameworks (e.g., Spark, Kafka, Flink) and cloud-native services (AWS) to support security analytics use cases
Drive CI/CD best practices, infrastructure automation, and performance tuning across distributed environments
Evaluate and pilot the use of AI/LLM technologies in data pipelines for anomaly detection, metadata enrichment, and automation
Ensure data security and privacy compliance across all data platforms and processes
Evaluate and integrate LLM-based automation and AI-enhanced observability into engineering workflows
Collaborate with data scientists, product managers, and business leaders to understand data needs and deliver solutions that drive business value
Oversee the design and implementation of various databases, including NoSQL databases like HBase and Cassandra

10 to 15 years of experience in big data architecture and engineering, including deep proficiency with the AWS cloud platform, with at least 3-5 years in a leadership or management role
Expertise in distributed systems and frameworks such as Apache Spark, Scala, Kafka, and Flink, with experience building production-grade data pipelines
Strong programming skills in Java for building scalable data applications
Hands-on experience with ETL tools and orchestration systems
Solid understanding of data modeling across both relational (PostgreSQL, MySQL) and NoSQL (HBase) databases, as well as performance tuning
Demonstrated experience with AWS services including Lambda functions, AWS Step Functions, and CloudFormation (CF)
Strong interpersonal and communication skills to effectively lead a team and collaborate with a diverse group of stakeholders
Bachelor's or Master's degree in Computer Science, Engineering, or a related field
Experience overseeing design and implementation of databases including HBase and Cassandra
Preferred: Experience integrating AI/ML or LLM frameworks (e.g., LangChain, LlamaIndex) into data workflows
Preferred: Experience implementing CI/CD pipelines with Kubernetes, Docker, and Terraform
Preferred: Knowledge of modern data warehousing (e.g., BigQuery, Snowflake) and data governance principles (GDPR, HIPAA)

Benefits

Tip: use these terms in your resume and cover letter to boost ATS matches.

big data architecturedata engineeringAWSApache SparkScalaKafkaFlinkJavaETL toolsdata modeling

leadershipmentoringcommunicationcollaborationinterpersonal skills

Bachelor's degree in Computer ScienceMaster's degree in Computer Science