Lead QA Engineer
Titan AI · Remote · United States ·
COMPENSATION
$140k – $165k
Listed on the employer's job board (Ashby)
- Lead QA Engineer
- Fully remote (US)
- AI/ML testing focus
- Python + pytest
- Build QA from scratch
You will establish Titan's first formal QA function, designing evaluation frameworks for small language models and agentic AI. As an individual contributor, you will write test cases, build regression baselines, and integrate quality gates into CI/CD pipelines before scaling the team.
The work focuses on ensuring AI outputs meet strict banking compliance and operational standards. You will handle end-to-end testing for backend APIs, document ingestion, and frontend surfaces while producing audit-ready artifacts for SOC 2 Type II compliance.
What you'll do
- Design and execute evaluation frameworks for LLM and agentic AI outputs
- Define behavioral contracts and own regression baselines for model behavior
- Write and maintain automated test suites for APIs and AI workflows
- Set up and enforce quality gates within CI/CD pipelines
- Triage production bugs and create reproduction cases and regression tests
- Produce test artifacts and documentation for SOC 2 Type II audits
- Collaborate with engineering on client-side validation and issue resolution
What you'll need
- Seven or more years of experience in software QA engineering
- At least two years personally testing AI or ML systems
- Fluency in Python and experience with pytest, Playwright, or Selenium
- Hands-on experience with RAGAS, DeepEval, LangSmith, or similar tools
- Ability to trace failures from application layer to infrastructure
- Knowledge of Azure, async systems, and REST APIs
- Experience integrating QA gates into CI/CD pipelines end to end
Nice to have
- Experience in fintech, banking, or regulated environments
- Familiarity with document processing pipelines
- Knowledge of multi-agent architectures and RAG validation
- Experience with observability tooling like Arize or Langfuse
Stack
- Python
- pytest
- Playwright
- Selenium
- RAGAS
- DeepEval
- LangSmith
- Azure
- REST APIs
- Arize
- Langfuse
How you'll work
- Fully remote within the United States
- Occasional travel to client sites and team offsites required
What you get
- Competitive base salary
- Meaningful equity package