Human Data Evals Lead (Remote/US/LATAM)
Anyone AI · Remote · Argentina - Fully Remote · Uruguay · Chile · Ecuador - Fully Remote · Mexico - Fully Remote · Peru - Fully Remote ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Fully remote (US/LATAM)
- Lead level position
- AI benchmarking focus
- Manage expert pools
- Fluent English required
You will lead Anyone AI’s Human Data Division, owning the end-to-end process from data proposals to pilot delivery. This role involves designing high-quality sample packages and benchmarks that demonstrate model capabilities in reasoning, coding, and tool use.
Working closely with subject-matter experts and lab partners, you will ensure all deliverables meet rigorous quality standards. You act as the primary operational contact for AI labs, turning initial requests into successful production engagements.
What you'll do
- Create data proposals and sample packages to win work from AI labs
- Design benchmarks with expert-verified ground truth and reasoning trajectories
- Implement rigorous QC structures including calibration layers and verifiers
- Recruit, brief, and calibrate subject-matter experts across STEM and coding
- Serve as direct point of contact for lab partners on calls and Slack
- Own pilot delivery from scoping and staffing to final quality control
What you'll need
- 5+ years in technical delivery, quality, or program management
- Recent experience in AI/ML data, model evaluation, or benchmarking
- Hands-on experience delivering evaluation work to AI labs or enterprise teams
- Deep understanding of frontier model evaluation metrics and rubrics
- Proven leadership in recruiting and managing expert pools or vendors
- Fluent English communication skills
Nice to have
- Spanish language proficiency
How you'll work
- Fully remote position
- Based in US or LATAM time zones