Machine Learning Engineer - ML Training Platform
Pluralis Research · Remote · San Francisco · Australia ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Fully remote (global)
- Multi-cloud infrastructure
- Distributed ML systems
- Visa sponsorship available
- Senior level role
Join Pluralis Research to engineer the platform behind Protocol Learning, enabling large model training across decentralized consumer hardware. You will design resource management systems and fault-tolerant infrastructure that operates without traditional datacenters, handling dynamic node churn and real-world network constraints.
What you'll do
- Design resource management systems for AWS, GCP, and Azure using infrastructure-as-code
- Handle dynamic scaling and state synchronization across hundreds of heterogeneous nodes
- Architect fault-tolerant infrastructure for distributed ML training and inference
- Implement GPU cluster management, checkpointing, and resilient retry strategies
- Build systems to simulate and handle real network conditions like latency and packet loss
- Manage node churn and ensure data flow across workers with varying connectivity
What you'll need
- Production experience with Pulumi, Terraform, or CloudFormation for multi-cloud deployments
- Proficiency with Docker, Kubernetes (EKS), and managing GPU workloads at scale
- Deep understanding of distributed training workflows including checkpointing and sharding
- Knowledge of decentralized networking concepts such as P2P and NAT traversal
- Strong Python engineering skills with asyncio, concurrency, and cloud SDKs
- Hands-on experience with observability tools like Prometheus and Grafana
- Professional-level written and spoken English proficiency
Nice to have
- Experience with foundation model pre-training, post-training, or reinforcement learning
- Background in proprietary, open-weight, or open-source AI labs
Stack
- Python
- Pulumi
- Terraform
- CloudFormation
- Docker
- Kubernetes
- EKS
- AWS
- GCP
- Azure
- Prometheus
- Grafana
How you'll work
- Fully remote with team members distributed globally
- Main teams located in Australia and North America
- Requires comfort working across multiple time zones
What you get
- Equity-heavy package with significant ownership opportunities
- High base salary component
- Optional full visa sponsorship for Australia or the US
- Relocation support provided if needed