Research Scientist (Remote/US/LATAM)
Anyone AI · Remote · Fully Remote · Uruguay · Chile · United Kingdom · Ecuador - Fully Remote · Mexico - Fully Remote · Colombia - Fully Remote · France · Peru - Fully Remote · Brazil · Argentina - Fully Remote ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Fully remote (US/LATAM)
- LLM Benchmarking
- Research Focus
- Publishing Support
- Senior Level
Anyone AI Labs seeks a Research Scientist to tackle the challenge of reliably measuring what frontier AI models can do. You will own the evaluation problem by designing original benchmarks, validating them against multiple models, and ensuring they remain informative as technology advances.
The role involves collaborating with subject-matter experts and acting as a technical liaison for AI labs. You will transform complex measurement questions into robust evaluation packages that withstand rigorous scrutiny from buyers and the scientific community.
What you'll do
- Design original benchmarks to measure frontier model capabilities accurately
- Address measurement challenges like validity, reliability, and contamination
- Build evaluation packages with expert-verified ground truth and rigorous QC
- Recruit and calibrate a pool of experts in coding, agents, and STEM
- Serve as technical point of contact for AI labs to define evaluation needs
- Deliver sample packages and manage pilots from request to completion
- Publish generalized work as public benchmarks or papers at top venues
What you'll need
- Background in ML evaluation or benchmarking with a track record of published work
- Deep expertise in LLM benchmarking, especially for code and agentic models
- Strong understanding of psychometrics, rubrics, and construct validity
- Interest in safety, capability elicitation, and robustness in evaluation
- Proven ability to maintain rigorous standards for teams or expert pools
- Comfort with the full research loop from framing questions to writing up results
- Fluent English communication skills
Nice to have
- Proficiency in Spanish
How you'll work
- Fully remote position
- Open to candidates in the US and Latin America
- Reports directly to the CEO