Member of Technical Staff - Inference
Prime Intellect · Remote · San Francisco ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Fully remote option
- LLM inference focus
- Visa sponsorship
- Senior level role
- Equity incentives
You will design and operate multi-tenant LLM serving platforms across cloud GPU fleets. The work involves optimizing inference frameworks, managing distributed systems, and ensuring high availability for large-scale AI models.
What you'll do
- Build multi-tenant LLM serving platforms for cloud GPU fleets
- Design placement algorithms for heterogeneous accelerators
- Implement failover and traffic shifting for system resilience
- Optimize model distribution and reduce cold-start times
- Integrate inference frameworks like vLLM or SGLang
- Tune parallelism and memory management for performance
- Develop reproducible performance suites for latency and throughput
- Establish CI/CD pipelines with performance gates
What you'll need
- 3+ years building large-scale ML or LLM services
- Experience with vLLM, SGLang, or TensorRT-LLM
- Familiarity with distributed serving infrastructure
- Deep understanding of KV-cache and batching strategies
- Proficiency in debugging CUDA, NCCL, and networking issues
- Strong skills in Python for backend services
- Experience with PyTorch for inference engine development
- Knowledge of AWS or GCP deployment patterns
- Ability to run infrastructure at scale using Kubernetes
Nice to have
- Familiarity with CUDA or Triton kernel development
- Proficiency in Rust or C++ for systems performance
- Experience with Kafka, Redis, or gRPC
- Knowledge of Prometheus, Grafana, or OpenTelemetry
- Use of Terraform or Ansible for infrastructure automation
- Contributions to open-source serving or RL projects
Stack
- Python
- PyTorch
- Kubernetes
- vLLM
- SGLang
- TensorRT-LLM
- CUDA
- NCCL
- AWS
- GCP
How you'll work
- Fully remote or San Francisco office
- Flexible work arrangement
What you get
- Salary range $150k-$300k
- Significant equity incentives
- Full visa sponsorship
- Relocation support
- Professional development budget
- Team off-sites and conference attendance