Senior Site Reliability Engineer
Andromeda Cluster · Remote · Global Remote / San Francisco, CA ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Senior SRE Role
- Fully Remote
- GPU Cluster Ops
- Kubernetes + Go
- AI Infrastructure
You will design, debug, and operate multi-region GPU compute clusters optimized for distributed training. Working directly with customers, you will troubleshoot real-time issues and optimize performance from the network fabric to the kernel level.
The team focuses on enabling fragmented global compute supply to flow through a single platform, ensuring reliable capacity for model builders and research labs.
What you'll do
- Design multi-region GPU clusters for large-scale training throughput
- Serve as primary technical contact for customer training workloads
- Define SLOs accounting for GPU-specific failure modes like ECC errors
- Ensure health of high-speed interconnects such as InfiniBand and NVLink
- Build deep observability into GPU utilization and hardware health
- Automate cluster provisioning, health checks, and driver lifecycle management
- Lead incident response for complex hardware and networking failures
What you'll need
- Hands-on experience operating large-scale NVIDIA GPU clusters
- Production experience with InfiniBand, RoCE, or NVLink fabrics
- Working knowledge of NCCL, CUDA, PyTorch distributed, or DeepSpeed
- Expert-level Linux knowledge including kernel tuning and driver management
- Strong experience running Kubernetes in production with GPU workloads
- Proficiency in Python, Go, or Bash for building production tools
- Experience building GPU-specific telemetry and monitoring dashboards
Nice to have
- Experience with parallel file systems like VAST, Weka, or Lustre
- Background in profiling and optimizing distributed training performance
- Involvement in physical cluster design and hardware validation
- Experience leading or mentoring infrastructure engineering teams
Stack
- NVIDIA A100/H100/B200
- InfiniBand
- RoCE
- NVLink
- NCCL
- CUDA
- PyTorch
- DeepSpeed
- Linux
- Kubernetes
- Slurm
- Python
- Go
- Bash
- Terraform
- Helm
- Ansible
- Prometheus
- Grafana
- DCGM
How you'll work
- Fully remote position available globally
- Full-time employment opportunity