Staff Research Engineer – RRI Harness
Sequen AI · Remote · United States ·
COMPENSATION
$300k – $350k
Listed on the employer's job board (Ashby)
- Staff level role
- Python + Go
- LLM agent systems
- Fully remote option
- GPU infrastructure
You will lead the development of the harness that powers Sequen's Recursive Ranking Intelligence engine. This system orchestrates teams of AI agents to perform ML research tasks in client clouds or hosted instances.
The position blends research engineering with infrastructure work. You will design experiments to improve agent performance, ensure system reliability over multi-day sessions, and build observability tools to monitor costs and behavior.
What you'll do
- Build and evolve the Python multi-agent runtime including orchestration and memory management
- Design and run experiments on prompting, tool design, and model choice to improve results
- Create fast evaluation sets and scorecards to measure noise and component ablations
- Ensure multi-day sessions survive GPU failures, node replacements, and provider errors
- Implement event streams and admin tooling for per-agent cost and token accounting
- Optimize code to reduce agent idling, looping, and unnecessary token consumption
- Define clean interfaces between the harness, Go control plane, and GPU proxy
- Collaborate with scientists and engineers to diagnose and fix production client issues
What you'll need
- Seven or more years of experience building production software with strong Python skills
- Proven history of working with distributed or long-running systems
- Experience building with LLMs in agentic loops including tool calling and streaming
- Deep understanding of how agentic systems fail at scale and how to mitigate risks
- Ability to debug across process, container, and network boundaries on Kubernetes
- Fluency in reading training scripts and distinguishing real regressions from noise
- Strong sense of ownership for system reliability and end-to-end observability
Nice to have
- Experience running ML or LLM experiments end to end with baselines and ablations
- Background in building test harnesses, benchmarks, or CI for ML systems
- Working knowledge of Go for contributions to the control plane and proxy
- Experience shipping software to on-prem, BYOC, or air-gapped environments
- Familiarity with PyTorch training at scale, GPU scheduling, or ML platforms
- Contributions to open-source agent frameworks, eval harnesses, or ML tooling
Stack
- Python
- Go
- Kubernetes
- PyTorch
- MCP
- LLMs
How you'll work
- Flexible hybrid or remote configurations available
- Unlimited paid time off offered
- Work involves GPU nodes and client cloud environments
What you get
- High-impact staff-level role owning core runtime
- Opportunity to build infrastructure for long-horizon AI agents
- Collaborative world-class engineering culture