Customer Reliability Engineer
Andromeda Cluster · Remote · Global Remote / San Francisco, CA ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Remote or SF Hybrid
- Linux + Kubernetes
- Python + Bash scripting
- 24/7 On-call rotation
- AI Infrastructure focus
You will serve as the primary technical contact when customer AI workloads fail on GPU clusters. Your work involves debugging Linux and Kubernetes layers, coordinating with external infrastructure providers to resolve faults, and creating automation to detect issues proactively.
This position requires strong troubleshooting skills and the ability to communicate clearly during incidents. You will join a 24/7 on-call rotation to ensure reliable compute capacity for model builders and research labs.
What you'll do
- Triage and resolve customer issues from diagnosis to closure
- Debug Linux layer problems including networking, storage, and kernel logs
- Investigate Kubernetes failures such as pending pods and scheduling errors
- Address GPU driver issues, thermal throttling, and device-plugin errors
- Participate in a 24/7 on-call rotation for incident response
- Coordinate with external compute providers to resolve infrastructure faults
- Develop Python or Bash scripts to automate health checks and monitoring
- Maintain runbooks and documentation for recurring technical issues
What you'll need
- Strong Linux troubleshooting skills via command line and logs
- Working knowledge of Kubernetes pods, nodes, and scheduling
- Ability to write automation scripts in Python or Bash
- Clear technical writing and communication skills for customers
- Sound judgment under pressure during critical incidents
- Willingness to join a 24/7 on-call rotation
Nice to have
- Experience with NVIDIA GPUs, CUDA, and DCGM in production
- Familiarity with high-performance networking like InfiniBand or RoCE
- Background in HPC or batch schedulers such as Slurm
- Previous customer-facing technical support or solutions experience
- Knowledge of monitoring tools like Prometheus, Grafana, or Datadog
- Experience with Infrastructure as Code using Terraform or Ansible
Stack
- Linux
- Kubernetes
- Python
- Bash
- NVIDIA GPUs
- CUDA
- Prometheus
- Grafana
- Terraform
How you'll work
- Fully remote or hybrid based in San Francisco
- Full-time employment
- Includes participation in a 24/7 on-call rotation
What you get
- Competitive compensation package
- Meaningful equity ownership
- Healthcare, dental, and vision coverage
- 401(k) retirement plan
- Unlimited paid time off