Senior ML Systems Engineer, Inference
Runpod · Remote · Remote - USA ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Senior level role
- Fully remote
- Python + vLLM/SGLang
- LLM inference focus
- No visa sponsorship
Join Runpod's remote-first team to make their AI Developer Cloud the fastest and most cost-efficient place for LLM inference. You will own serving performance end-to-end, measuring and improving latency and throughput across various models and hardware generations. Your work will directly impact the experience of customers running frontier models in production.
What you'll do
- Define and build tooling for rigorous inference performance measurement
- Profile and diagnose issues across scheduling, memory, and kernels
- Improve serving efficiency for large models on single and multi-node GPUs
- Convert findings into production-ready runtimes and default configurations
- Collaborate with product and infrastructure teams on inference offerings
- Evaluate and adopt relevant open-source inference ecosystem developments
- Trace and fix bottlenecks in the serving engine when tuning is insufficient
What you'll need
- 5+ years of professional system engineering experience
- Deep hands-on experience with vLLM, SGLang, or similar serving engines
- Strong Python skills for large, performance-critical codebases
- Solid understanding of batching, memory, parallelism, and latency trade-offs
- Experience with quantization, speculative decoding, or distributed serving
- Rigor in benchmarking and comfort with GPU profiling tools
- Ability to clearly explain results and drive decisions based on data
Nice to have
- Experience writing or tuning GPU kernels in CUDA or Triton
- Contributions to inference or ML systems open-source projects
- Experience with multi-node GPU systems and high-speed networking
- Background in companies where inference cost and latency were core metrics
Stack
- Python
- vLLM
- SGLang
- CUDA
- Triton
How you'll work
- Fully remote work policy
- Slack used for internal communication
- Inclusive and collaborative team culture
What you get
- Competitive base salary between $150k and $220k
- Meaningful equity and stock options for all employees
- Generous medical, dental, and vision plans
- Flexible paid time off policy
- $1,200 home office and equipment stipend