QA Analyst – LOIS for Word

Filevine

QA Analyst owning the quality for LOIS for Word, Filevine's AI-powered legal drafting product. Building and owning QA processes including strategy, testing, and automation.

Posted 6/2/2026full-timeRemote • California • 🇺🇸 United StatesJuniorMid-Level💰 $105,000 - $150,000 per yearWebsite

About the role

Key responsibilities & impact

Define and own the test strategy, process, and QA playbook for the product
Move QA upstream by partnering with product and engineering early, so test plans are ready before features ship
Evaluate AI output quality across dimensions like accuracy, citation reliability, and appropriately scoped reasoning
Design and maintain evaluation sets that account for non-deterministic output and catch regressions over time
Test how AI-generated edits apply to documents, including accuracy and formatting fidelity across document structures
Build automated guardrails integrated into CI/CD that cover critical flows
Write structured, reproducible bug reports
Partner with legal domain experts to understand how different customer types use the product, and translate that into test coverage
Approach QA with a product and design mindset. If something works but could be better, we want to hear it
Compare behavior across product surfaces and document discrepancies

Requirements

What you’ll need

2+ years of QA experience, including ownership of test strategy for a product or major feature area
Track record of building QA process and automation from scratch in a fast-moving environment
Comfort operating in high ambiguity, where features ship quickly and you figure out how to test them fast
Strong analytical and problem-solving skills, with attention to both functional correctness and AI output quality
Ability to reason about non-deterministic systems and define what "correct" means when there's no single right answer
Excellent written communication, with the ability to write clear, concise, actionable bug reports backed by evidence
Experience with bug tracking tools and test management workflows

Benefits

Comp & perks

Medical, Dental, & Vision Insurance (for full-time employees)
Competitive & Fair Pay
Maternity & paternity leave (for full-time employees)
Short & long-term disability
Opportunity to learn from a dedicated leadership team
Top-of-the-line company swag

ATS Keywords

✓ Tailor your resume

Applicant Tracking System Keywords

Tip: use these terms in your resume and cover letter to boost ATS matches.

Hard Skills & Tools

QA processtest strategyautomationbug reportingAI output evaluationnon-deterministic systemstest plansCI/CD integrationdocument accuracyregression testing

Soft Skills

analytical skillsproblem-solvingattention to detailwritten communicationcollaborationadaptabilityproduct mindsetdesign mindsetcritical thinkingambiguity management