Research Scientist, Benchmarks & Evals
Kled AI · Remote · United States ·
COMPENSATION
$250k – $350k
Listed on the employer's job board (Ashby)
- Fully remote
- Senior level (5-7+ years)
- PyTorch + Python
- High equity package
- Benchmark design focus
You will build the benchmarks that validate the quality of Kled's human data network for leading AI laboratories. The work includes designing public benchmarks, creating private test sets from opt-in data, and running experiments to measure model performance improvements.
Your findings will directly influence data collection strategies and licensing deals. You will collaborate with founders and lab buyers to translate technical results into business value, ensuring researchers trust your leaderboards and reports.
What you'll do
- Design and release public benchmarks for image, video, and document models
- Build private test sets using real, opt-in data unseen by trained models
- Conduct data-value experiments to measure impact on model training
- Create human evaluation studies and corresponding automatic metrics
- Publish trusted leaderboards and reports for the research community
- Identify model weaknesses to guide future data collection efforts
- Collaborate with founders and buyers to convert results into data deals
What you'll need
- 5-7+ years of experience in ML research or research engineering
- Proven track record of building benchmarks used by others
- Strong hands-on experience training and fine-tuning models with PyTorch
- Deep knowledge of experimental design, statistics, and ablations
- Ability to run end-to-end experiments from GPU setup to final reporting
- Clear writing skills tailored for both researchers and business buyers
Nice to have
- Publications at NeurIPS, ICML, ICLR, or CVPR
- Experience on an evaluation or data team at an AI lab
- Background in selling or licensing data to AI laboratories
- Research experience in data valuation or attribution
Stack
- Python
- PyTorch
- PostgreSQL
- Supabase
- S3
- Cloud GPUs
What you get
- Base salary between $250,000 and $350,000
- Equity package ranging from $750,000 to $1,500,000
- Opportunity to own core systems at scale
- Work with a team experienced in marketplaces and AI systems