Senior Software Engineer - Model Evaluation & AI Systems
Deepgram · Remote · USA - Remote ·
COMPENSATION
$180k – $240k
Listed on the employer's job board (Ashby)
- Senior Software Engineer
- Fully remote role
- AI & Voice Tech
- Python, Rust, or Go
- Model Evaluation Focus
Join the team responsible for validating the quality of Deepgram's speech-to-text, text-to-speech, and multimodal models. You will design and maintain the evaluation surfaces that catch regressions and hallucinations before they impact customers.
The role involves building scalable infrastructure to measure model performance in batch and streaming environments. You will partner with Research to turn benchmarks into automated checks and provide trusted signals for release decisions.
What you'll do
- Define evaluation methodologies for STT, TTS, LLM, and multimodal systems
- Build automated pipelines for metrics like WER, latency, and hallucination detection
- Create scalable infrastructure for running tests on production models and GPU clusters
- Translate research benchmarks into automated pass/fail gates for releases
- Operate canaries and monitoring systems to detect quality regressions early
- Integrate quality gates into CI/CD workflows for continuous verification
- Partner with DevOps to manage ephemeral test environments and result aggregation
- Collaborate with Research and Product teams to inform optimization decisions
What you'll need
- BS, MS, or PhD in Computer Science, AI, Applied Math, or equivalent experience
- 5+ years of software or QA engineering experience shipping test infrastructure
- Strong backend or scripting skills in Python, Rust, Go, or similar languages
- Experience designing automated test pipelines or data-processing systems
- Ability to analyze metrics and distinguish real regressions from statistical noise
- Comfortable taking charge of ambiguous technical challenges
- Effective communication skills across research, engineering, and product teams
Nice to have
- Experience evaluating LLMs, RAG pipelines, agents, or multimodal models
- Familiarity with React Native or cross-platform mobile frameworks for tooling
- Background in voice, audio, or speech recognition metrics like WER and MOS
- Experience building evaluation frameworks used by external users or other teams
- Knowledge of cloud infrastructure, containers, and monitoring tools like Grafana
- Prior involvement in open-source projects through contributions or maintenance
Stack
- Python
- Rust
- Go
- React Native
- Grafana
How you'll work
- Fully remote position
- AI-first mindset required for daily workflows
- Fast-paced environment with rapid technological changes
- Expectation to actively use and experiment with advanced AI tools