Principal Production Engineer

Zscaler

Principal Production Engineer at Zscaler developing scalable infrastructure and driving automation culture. Ensuring reliability of a global platform processing over 200 billion transactions daily.

Posted 6/16/2026full-timeSan Jose • California • 🇺🇸 United StatesLead💰 $164,500 - $235,000 per yearWebsite

Tech Stack

Tools & technologies

AWSGoGoogle Cloud PlatformGrafanaLinuxPrometheusPython

About the role

Key responsibilities & impact

Design and implement highly available, scalable infrastructure across AWS, GCP, and bare-metal environments
Drive an "automation-first" culture by writing code (Python/Go) to eliminate manual toil and build self-healing systems
Implement and maintain sophisticated observability (Prometheus, Grafana, OpenTelemetry), define SLIs/SLOs, and establish error budgets
Act as a lead Incident Commander (TDO on-call), develop response playbooks, and conduct deep-dive post-incident analyses
Partner with Engineering and partner teams to conduct operability reviews

Requirements

What you’ll need

10+ years of experience managing reliability, scalability, and availability for large-scale production services
Deep expertise in programming (e.g., Python, Go, or C/C++)
Strong background in networking protocols, Linux/RHEL systems, and distributed architecture
Experience in high-stakes incident management and participation in a 24/7 on-call rotation
Proficiency in leveraging ITIL frameworks and incident data to drive service maturity through systematic problem management and technical operability reviews

Benefits

Comp & perks

Various health plans
Time off plans for vacation and sick time
Parental leave options
Retirement options
Education reimbursement
In-office perks, and more!

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonGoC/C++AWSGCPPrometheusGrafanaOpenTelemetryLinuxRHEL

Soft Skills

leadershipincident managementcommunicationcollaborationproblem management