Senior Business Intelligence Analyst – AI Evaluation, Monitoring

Geisinger

Senior BI Analyst supporting AI Evaluation & Monitoring, utilizing analytic expertise for data-driven decisions. Collaborating with stakeholders and mentoring junior BI developers in a healthcare setting.

Posted 5/6/2026full-timeRemote • Pennsylvania • 🇺🇸 United StatesSeniorWebsite

Tech Stack

Tools & technologies

NumpyPandasPythonScikit-LearnSQL

About the role

Key responsibilities & impact

The Senior BI Analyst serves as a partner to the business or clinical area supported (AI Evaluation & Monitoring)
Use analytic expertise including knowledge of data structures, programming, and statistics with critical thinking
Collect, clean, and interpret data sets to answer a question or solve a problem
Identify and effectively communicate improvement opportunities and support evaluation of existing AI/ML projects
Act as the analytics expert in meetings and discussions with stakeholders
Consult with AI stakeholders to gather evaluation and data requirements and define scope on projects of all sizes
Execute the evaluation work that keeps Geisinger's AI portfolio safe, performant, and accountable
Collaborate with internal partners and organizations in design and development of reporting solutions and analytic studies
Develop, validate, and execute complex queries and reports that answer business questions for internal customers
Present analysis and reports to internal customers and stakeholders
Report and help troubleshoot data quality issues during data profiling, testing, or validation
Participate in strategic departmental initiatives and mentor junior BI developers

Requirements

What you’ll need

Minimum of 8 years-Relevant experience* (Required)
Healthcare, clinical, or regulated-industry data experience strongly preferred
Applied statistics: hypothesis testing, confidence intervals, power and sample size, regression, and non-parametric methods
SQL with large, messy clinical data warehouses, including window functions, CTEs, debugging query plans, and performance tuning
Python for analysis: pandas, numpy, scikit-learn, statsmodels, common evaluation libraries
Production monitoring design and execution: drift detection, performance decay tracking, adoption and outcome metrics
LLM and generative AI evaluation execution: golden sets, judge-based scoring, hallucination and grounding checks
Education: High School Diploma or Equivalent (GED)- (Required)

Benefits

Comp & perks

Healthcare benefits for full time and part time positions from day one, including vision, dental and domestic partners
Encouragement of collaboration, cooperation and collegiality

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

SQLPythonapplied statisticshypothesis testingconfidence intervalsregressionnon-parametric methodsdata cleaningdata interpretationcomplex queries

Soft Skills

critical thinkingcommunicationcollaborationmentoringstakeholder engagementproblem-solvinganalytical thinkingpresentation skillsconsultationstrategic initiative participation