Senior/Staff/Principal Software Engineer – Observability Engineering

AppGate

Observability Engineer at AppGate designing solutions for high-performance telemetry and data infrastructures. Leading technical decisions and collaborating with cross-functional teams to enhance observability capabilities.

Posted 5/11/2026full-timeRemote • New York • 🇺🇸 United StatesSeniorWebsite

Tech Stack

Tools & technologies

DNSGoGrafanaPrometheusRustSplunk

About the role

Key responsibilities & impact

Own the end-to-end design and implementation of the AppGate observability fabric — from telemetry SDKs in our clients and gateways, to the LogForwarder pipeline, to customer-side integrations.
Make foundational technical decisions — transport protocols, sampling strategies, schema design, correlation models — that determine whether our platform scales gracefully to hundreds of millions of events per day.
Enable next-generation capabilities, including OpenTelemetry-Native Telemetry Fabric, High-Cardinality Data Pipeline, End-to-End Distributed Tracing, On-Demand Packet Capture, and more.
Define telemetry schema, correlation model, transport, and sampling strategies spanning client devices, controllers, and gateways.
Validate at Customer Scale: Test in lab environments matching our largest deployments and hunt down cardinality explosions and pipeline backpressure before customers see them.
Drive Integration Standards: Own the OTLP, Prometheus, and JSON-log compatibility surface and validate ingestion into Datadog, Splunk, Nexthink, and Elastic.
Collaborate Cross-Functionally: Work directly with product, R&D, and marquee customers in defense and critical infrastructure to shape requirements and deliver outcomes that matter.

Requirements

What you’ll need

8+ years of engineering experience with at least 4 years dedicated to observability, telemetry, or large-scale data infrastructure (Datadog, Splunk, Elastic, Honeycomb, New Relic, Grafana Labs, or equivalent).
Deep OpenTelemetry expertise: OTLP, the OTel Collector, semantic conventions, context propagation, and head/tail sampling — you can debate the trade-offs in your sleep.
Distributed tracing in production: You’ve designed or significantly contributed to a tracing system handling real customer traffic, not just a side project.
High-throughput pipeline experience: Hands-on with systems ingesting 100M+ events per day, including back-pressure handling, batching, and storage trade-offs.
Strong systems programming: Production Go and/or Rust preferred. Comfort across the stack, from agent code to backend services.
Networking and security fluency: Comfortable with TLS, DNS, TCP, and identity protocols. Prior ZTNA, SASE, or SD-WAN experience is a strong plus.
Mindset: Pragmatic, opinionated, and impact driven. You know when to prototype and when to ship.

Benefits

Comp & perks

🌐 Worldwide ❌ Jobs You've Hidden ⭐️ Saved Jobs ✅ Applied Jobs ✉️ Email Alerts 👤 Account AppGate Website LinkedIn All Job Openings 501 - 1000 employees 🔒 Cybersecurity 🏢 Enterprise Cybersecurity
Enterprise AppGate is a global cybersecurity company that delivers high-performance Zero Trust Network Access (ZTNA) solutions for enterprises and government agencies. Its platform enforces identity-based, adaptive access policies using real-time risk scoring, AI-powered application discovery, and a direct-routed architecture designed to avoid cloud bottlenecks and scale with demanding environments. AppGate also provides professional services and cyber advisory offerings — including adversary simulation, penetration testing, and third-party access risk assessments — to help organizations implement and operationalize Zero Trust controls. Senior/Staff/Principal Software Engineer – Observability Engineering 🔥 5 minutes ago 🗽 New York – Remote ⏰ Full Time 🟠 Senior 🖥 Software Engineer DNS Grafana Prometheus Rust Splunk Go Apply Now Find Hiring Managers Customize resume + cover letter Report problem ☆ Save ☑️ Mark as applied ❌ Hide 📋 Description
Own the end-to-end design and implementation of the AppGate observability fabric — from telemetry SDKs in our clients and gateways, to the LogForwarder pipeline, to customer-side integrations.
Make foundational technical decisions — transport protocols, sampling strategies, schema design, correlation models — that determine whether our platform scales gracefully to hundreds of millions of events per day.
Enable next-generation capabilities, including OpenTelemetry-Native Telemetry Fabric, High-Cardinality Data Pipeline, End-to-End Distributed Tracing, On-Demand Packet Capture, and more.
Define telemetry schema, correlation model, transport, and sampling strategies spanning client devices, controllers, and gateways.
Validate at Customer Scale: Test in lab environments matching our largest deployments and hunt down cardinality explosions and pipeline backpressure before customers see them.
Drive Integration Standards: Own the OTLP, Prometheus, and JSON-log compatibility surface and validate ingestion into Datadog, Splunk, Nexthink, and Elastic.
Collaborate Cross-Functionally: Work directly with product, R&D, and marquee customers in defense and critical infrastructure to shape requirements and deliver outcomes that matter. 🎯 Requirements
8+ years of engineering experience with at least 4 years dedicated to observability, telemetry, or large-scale data infrastructure (Datadog, Splunk, Elastic, Honeycomb, New Relic, Grafana Labs, or equivalent).
Deep OpenTelemetry expertise: OTLP, the OTel Collector, semantic conventions, context propagation, and head/tail sampling — you can debate the trade-offs in your sleep.
Distributed tracing in production: You’ve designed or significantly contributed to a tracing system handling real customer traffic, not just a side project.
High-throughput pipeline experience: Hands-on with systems ingesting 100M+ events per day, including back-pressure handling, batching, and storage trade-offs.
Strong systems programming: Production Go and/or Rust preferred. Comfort across the stack, from agent code to backend services.
Networking and security fluency: Comfortable with TLS, DNS, TCP, and identity protocols. Prior ZTNA, SASE, or SD-WAN experience is a strong plus.
Mindset: Pragmatic, opinionated, and impact driven. You know when to prototype and when to ship. Apply Now 📊 Check your resume score for this job Improve your chances of getting an interview by checking your resume score before you apply. Check Resume Score Similar Jobs Desenvolvedor NodeJs 🔥 7 minutes ago Grupo Bimbo 10,000+ employees 👥 B2C 🛒 Retail 🌾 Agriculture Website LinkedIn All Job Openings Desenvolvedor NodeJs na Gupy, ajudando com processos internos. Envolvimento com validação de testes, automação de tarefas e codificação. 🇺🇸 United States – Remote ⏰ Full Time 🟡 Mid-level 🟠 Senior 🖥 Software Engineer 🗣️🇧🇷🇵🇹 Portuguese Required Node.js Developer 🔥 7 minutes ago Grupo Bimbo 10,000+ employees 👥 B2C 🛒 Retail 🌾 Agriculture Website LinkedIn All Job Openings Desenvolvedor NodeJs na Gupy, uma startup que melhora processos internos. Ajudar na validação de testes, automatização de tarefas e codificação. 🇺🇸 United States – Remote ⏰ Full Time 🟡 Mid-level 🟠 Senior 🖥 Software Engineer 🗣️🇧🇷🇵🇹 Portuguese Required Node.js Developer 🔥 7 minutes ago Grupo Bimbo 10,000+ employees 👥 B2C 🛒 Retail 🌾 Agriculture Website LinkedIn All Job Openings Desenvolvedor NodeJs colaborando em validação de testes e automação de tarefas no time de desenvolvimento da GUPY. Codificando e ajudando em deploys com horário flexível. 🇺🇸 United States – Remote ⏰ Full Time 🟡 Mid-level 🟠 Senior 🖥 Software Engineer 🗣️🇧🇷🇵🇹 Portuguese Required Analista Programador, SQL 🔥 7 minutes ago Grupo Bimbo 10,000+ employees 👥 B2C 🛒 Retail 🌾 Agriculture Website LinkedIn All Job Openings Analista Programador SQL em startup focada em melhorar processos internos e gestão de KPIs. Responsável por interação com usuários e candidatos para assegurar serviço humanizado. 🇺🇸 United States – Remote ⏰ Full Time 🟡 Mid-level 🟠 Senior 🖥 Software Engineer 🗣️🇧🇷🇵🇹 Portuguese Required Node.js Developer 🔥 7 minutes ago Grupo Bimbo 10,000+ employees 👥 B2C 🛒 Retail 🌾 Agriculture Website LinkedIn All Job Openings Desenvolvedor NodeJs at Gupy helping improve team development processes and support daily tasks with programming and site creation. 🇺🇸 United States – Remote ⏰ Full Time 🟡 Mid-level 🟠 Senior 🖥 Software Engineer 🗣️🇧🇷🇵🇹 Portuguese Required View More Software Engineer Jobs 🌐 Worldwide Built by Lior Neu-ner. I'd love to hear your feedback — Get in touch via DM or support@remoterocketship.com Search Search Jobs by country Search jobs by city Search jobs by job title Search entry-level jobs Search junior-level jobs Search senior-level jobs Search jobs by tech stack Search jobs by contract type Search remote internships Search remote part-time jobs Remote jobs Anywhere in the World Companies Hiring Anywhere in the World Companies Hiring Sales People Anywhere in the World Companies Hiring Software Engineers Anywhere in the World Resources Advice Tips for finding remote jobs Interview questions and answers Resume examples Cover letter examples Post a job Affiliates Privacy policy Terms of service Job board SEO course AI Apply Copilot OpenClaw job finder Jobs by Country Remote jobs anywhere in the world (Worldwide remote jobs) Remote jobs United States Remote jobs Australia Remote jobs Brazil Remote jobs Canada Remote jobs France Remote jobs Ireland Remote jobs Germany Remote jobs Netherlands Remote jobs Spain Remote jobs UK Popular Jobs Remote data analyst jobs Remote customer support jobs Remote executive assistant jobs Remote marketing jobs Remote product designer jobs Remote product manager jobs Remote project manager jobs Remote recruiter jobs Remote sales jobs Remote software engineer jobs Jobs by Type Remote full-time jobs Remote part-time jobs Remote contract jobs Remote internship jobs Remote entry-level jobs Remote jobs with no experience required Remote junior jobs (1-3 years of experience) Digital nomad jobs Remote jobs with no degree required Freelance remote jobs Temporary remote jobs Remote jobs hiring now Stay at home mom jobs

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

observabilitytelemetrylarge-scale data infrastructureOpenTelemetrydistributed tracinghigh-throughput pipelinesystems programmingGoRustnetworking

Soft Skills

pragmaticopinionatedimpact drivencollaboration