Design and build a **Streaming Transformation** process to evolve an existing scheduled PySpark ETL batch transformation process.
Reduce the time to ingest incoming data to where it shows up in our SaaS app & reports e2e, from hours to minutes or even seconds.
Produce effective design documentation to help circulate design ideas & transition plans. Break down approved designs into granular tasks.
Lead the implementation and take on the most challenging or escalated tasks personally while delegating less strategic work to a mix of onshore and offshore team members.
Meet regularly with technical and business stakeholders to communicate the plan, status, risks and delivery timeline estimates.
Drive the go-live of the streaming transformation and establish observability & operational SOP process to hand off to other engineers.
After go-live handle any escalated operational issues based on severity within a reasonable timeline and take on enhancement requests to harden the system and prevent future issues.
Foster innovation by designing and building solutions that solve business problems and by partnering with cross functional stakeholders and at senior levels.
Develop secure and high-quality production code. Review / debug code written by others.
Actively oversee live operations of our SaaS offering, monitoring KPI’s to find opportunities and threats.
Organize work around a cross-functional team of Product Managers, BAs, Engineers, BI Developers and Quality Engineers.
Stay up-to-date with the latest technologies and industry trends, leveraging them to drive platform innovation.
**Participate in week-long quarterly planning and roadmap meetings in-person.**
(For many cities) Participate in local office gatherings 1x every 2 months for Town Hall.

Requirements

5 years leading other software engineers
Expertise in data architecture and business intelligence systems including ELT development, data transformation, data modeling, report design / development and BI dashboard design / development.
10+ years of professional software engineering experience
5 years of Python experience
5 years of Typescript experience
5 years of SQL experience including complex joins, query performance, triggers, views
5 years of experience with Infrastructure as Code tooling, preferably CDK
Deep expertise & certifications on AWS
**Preferred:**
Experience with ETL tooling on AWS like PySpark, Lake Formation and Glue
Familiarity with Golang
Experience with Kinesis data streaming
Experience working with visualization tools like Tableau
Familiarity with HIPAA/SOC/HITRUST requirements

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonTypescriptSQLETLPySparkInfrastructure as CodeAWSKinesisTableauData modeling

Soft Skills

LeadershipCommunicationCollaborationProblem-solvingInnovationDocumentationTask delegationOperational oversightStakeholder engagementTeam organization

Certifications

AWS certification