Manager, Data Engineer

Arch Capital Group Ltd.

Manager, Data Engineer involved in end-to-end delivery of data solutions and analytics at Arch Capital. Leading a team and shaping scalable data strategies with a focus on AI transformation.

Posted 5/28/2026full-timeRemote • New Jersey, New York, North Carolina, Pennsylvania • 🇺🇸 United StatesMid-LevelSenior💰 $100,500 - $174,000 per yearWebsite

Tech Stack

Tools & technologies

ApacheAzureCloudPySparkPythonSparkSQL

About the role

Key responsibilities & impact

Lead delivery of high-quality data solutions by partnering with stakeholders and coaching data engineers.
Own end-to-end data engineering delivery across the project lifecycle.
Build strong partnerships across the organization to align priorities and deliver data-related goals.
Design clear, analytics-ready data structures by anticipating downstream analytical needs.
Evaluate and adopt new technologies and data sources to improve capability and efficiency.
Automate data ingestion and integration to reliably connect internal and external data sources.
Document data sources, definitions, and technical solutions to support transparency and reuse.
Reinforce strong delivery hygiene (version control, code review, automated testing, CI/CD, and operational readiness/monitoring).
Apply agentic, AI-assisted coding practices to accelerate delivery while maintaining appropriate controls.
Build agent-ready data assets, including semantic layer components (ontology, taxonomy, domain models) and governed access.
Provide retrieval-ready context (RAG pipelines, vector stores, knowledge bases) when needed.

Requirements

What you’ll need

6+ years’ experience of hands-on development in Python and distributed processing environments (e.g., Spark)
2–3+ years of technical leadership or project delivery ownership experience
Hands-on Databricks experience highly preferred
Strong programming expertise in Python and SQL, including data engineering frameworks, large-scale data manipulation, and governed AI-assisted development practices
Apache Spark proficiency (PySpark preferred) and experience with distributed data processing, including building scalable pipelines and optimizing performance for large-scale datasets
Cloud data platform proficiency (Snowflake, Databricks, Azure ecosystem fundamentals)
Data warehousing and modeling fundamentals (schema design, conformed definitions, performance optimization)
Data quality and observability practices (testing, reconciliation, monitoring)
Analytics & AI Readiness
Insurance data modeling for analytics and actuarial-ready data structures
MLOps familiarity supporting operationalized analytics and models
Semantic modeling skills (business definitions, metrics, ontology/taxonomy/domain models)
Business definition standardization for reuse across BI and AI use cases.

Benefits

Comp & perks

multiple medical plans plus dental, vision and prescription drug coverage
a competitive 401k with generous matching
PTO beginning at 20 days per year
up to 12 paid company holidays per year plus 2 paid days of Volunteer Time Offer
basic Life and AD&D Insurance as well as Short and Long-Term Disability
Paid Parental Leave of up to 10 weeks
Student Loan Assistance and Tuition Reimbursement
Backup Child and Elder Care
and more.

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

PythonSQLApache SparkDatabricksdata engineering frameworksdata warehousingdata modelingMLOpssemantic modelingautomated testing

Soft Skills

technical leadershipproject delivery ownershipstakeholder partnershipcoachingcommunicationcollaborationproblem-solvingorganizational skillsdelivery hygienetransparency