Research Scientist, Human Data & Evaluation
Contra · Remote · San Francisco ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Onsite in San Francisco
- Human-Centered AI Research
- Python and SQL analysis
- Multimodal evaluation
- $150k-$200k + Equity
Contra Labs builds creative preference datasets that power benchmarking and post-training for leading AI models. As a Research Scientist, you will translate ambiguous client questions into rigorous studies, defining what 'good' looks like across design, video, and imagery. You will own the full research lifecycle, from proposal development to final insight generation, helping shape the next generation of human creativity tools.
What you'll do
- Translate ambiguous client questions into clear research hypotheses and study designs
- Develop research proposals and methodologies from scratch without predefined approaches
- Design human-data collection strategies, evaluation protocols, and annotation schemes
- Create prompt strategies to elicit and compare model behavior across creative domains
- Analyze qualitative and quantitative data to explain patterns and generate recommendations
- Produce client-ready reports, presentations, benchmarks, and research publications
- Own research engagements independently from initial scoping through final delivery
What you'll need
- 3–5+ years of relevant research experience in HCI, Human-Centered AI, or ML
- Master’s, PhD, or equivalent industry research depth in a related field
- Strong skills in human evaluation, experimentation, and data collection
- Ability to independently design and execute rigorous research projects
- Comfortable analyzing data using Python and/or SQL
- Strong written communication skills to explain methodological tradeoffs
- Based in San Francisco, California
Nice to have
- Familiarity with modern ML and generative AI systems
- Understanding of RLHF, post-training, and preference data
- Experience studying image, video, multimodal, or agentic systems
- Experience designing prompts or systematically studying model behavior
- Background in working with large-scale human-data or annotation programs
- Strong publication record or evidence of original research ideas
Stack
- Python
- SQL
How you'll work
- Onsite role based in San Francisco, California
What you get
- Salary range of $150,000 to $200,000 USD plus equity
- Medical, Dental, and Vision benefits
- 401k matching program
- Company laptop provided on start date
Hiring process
- Interview with Recruiting Team (20 minutes)
- Interview with Head of Research (30 minutes)
- Technical interview with Data Lead (30 minutes)
- Interview with Head of Contra Labs (30 minutes)
- Paid Case Study and Presentation (60 minutes)