Senior Software Engineer, GPU Cluster Infrastructure
FAR.AI · Remote · Remote (International) ·
COMPENSATION
$150k – $275k
Listed on the employer's job board (Ashby)
- Senior Infrastructure Engineer
- Kubernetes + GPU Clusters
- Fully Remote (Global)
- AI Safety Research Focus
- Visa Sponsorship Available
Join the Foundations team at FAR.AI to build and operate the compute platform powering advanced AI safety research. You will manage a rapidly growing fleet of bare-metal and managed Kubernetes GPU clusters, ensuring performance and fault tolerance for frontier models.
The role involves direct collaboration with researchers to scale experiments, automate workflows, and harden security for AI agents. You will own infrastructure as code, batch scheduling, and distributed storage systems while helping define the future of the infrastructure sub-team.
What you'll do
- Operate Kubernetes GPU fleet including node lifecycle, upgrades, and capacity planning
- Manage batch scheduling, multi-tenancy, quotas, and fair share across research teams
- Design and run high-performance shared filesystems and object storage tiers
- Ensure fault tolerance for multi-node training runs and debug network fabric issues
- Harden platform security covering identity, access, and workload isolation
- Integrate new hardware providers and acceptance-test fabric and storage throughput
- Collaborate with researchers to resolve infrastructure problems and improve platform
What you'll need
- 3+ years in systems or infrastructure engineering on production Linux environments
- Experience running production Kubernetes for GPU workloads with batch layers
- Proficiency with infrastructure as code tools like Terraform or Ansible
- Strong programming skills in Python, Go, Rust, or C++ for automation services
- Ability to write clear technical documentation for engineers and researchers
- Experience with observability tools such as Prometheus, Helm, or ArgoCD
Nice to have
- Deep knowledge of distributed training infrastructure and NCCL debugging
- Experience with distributed storage systems like VAST, Weka, Lustre, or Ceph
- Expertise in cluster security, sandboxed runtimes, and admission control
- Familiarity with Kubernetes scheduler plugins and custom controllers
- Experience managing multi-provider platforms with uneven data locality
Stack
- Kubernetes
- Linux
- Python
- Go
- Rust
- C++
- Terraform
- Ansible
- Helm
- ArgoCD
- Prometheus
- PyTorch
- NCCL
- Slurm
- Kueue
- Volcano
How you'll work
- Full-time position, 40 hours per week
- Fully remote options available in most countries
- Preference for working hours overlapping with Berkeley, CA
- In-person options available in Berkeley, CA or Singapore
- Informal incident coverage during working hours currently
- Formal on-call rotation expected to be introduced in future
What you get
- 94% of health insurance premium paid by organization (US)
- 401(k) plan with up to 2% employer match (US)
- 25 days paid time off per year plus sick leave (US)
- Paid bereavement, family, medical and pregnancy disability leave
- Work computer and WFH stipend provided for eligible employees
- Visa sponsorship available for in-person employees
- EOR employment structure for international staff outside US/Singapore