Design and implement extremely scalable large scale distributed Spark cluster on Kubernetes platform optimized for Data center environment.
Perform and oversee a variety of functions to ensure that our Spark infrastructure on Kubernetes is available, reliable, stable and secure
Applies judgment in analyzing and selecting technologies, installation and maintenance of software and hardware systems that allow multiple Engineering and DataScience teams to interact with this system
Design, writes and own APIs, automate and generate reports to visualize cluster health.
Has responsibility for building and maintaining the Spark infrastructure such that software engineers, data analysts and data scientists can run jobs to gather data and insights.
Work on Query systems, like Apache Presto or Trino, Jupyter Notebooks

Requirements

Strong in one or both of field i.e., Spark ecosystem or Kubernetes ecosystem
Strong analytical skills and with deep understanding of Distributed Systems
Strong programming skills in languages as Go, Java or Python
Understanding of Apache Spark ecosystem technologies (Deltalake/Iceberg, YARN, HDFS, Kafka, Zookeeper, Hive, Jupyter notebooks)
Experience with large-scale business critical platforms with Spark on Kubernetes on data center or cloud.
Experience with continuous deployment on K8 with Helm, fluxCD etc
Solid understanding of Spark(data) and K8 storage systems (Object stores S3/Minio, Spark ephemeral storage, Persistent volumes mapping and claims)
Understanding or spark memory management and experience with spark internals.
Familiarity with Chef is preferred.
Proven ability to work with both local and remote teams
Strong communication skills both verbal and written.

Benefits

Market leader in compensation and equity awards
Comprehensive physical and mental wellness programs
Competitive vacation and holidays for recharge
Paid parental and adoption leaves
Professional development opportunities for all employees regardless of level or role
Employee Networks, geographic neighborhood groups, and volunteer opportunities to build connections
Vibrant office culture with world class amenities
Great Place to Work Certified™ across the globe

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

Apache SparkKubernetesGoJavaPythonDeltalakeIcebergYARNHDFSKafka

Soft Skills

analytical skillscommunication skillsteam collaboration