Data Annotation Specialist, Generalist
Cohere · Remote · Canada · Calgary · Winnipeg · Vancouver · Halifax · Edmonton · Toronto · Victoria · Montreal ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Part-time (16 hrs/week)
- Remote (Canada only)
- Independent Contractor
- CAD $30/hour
- English proficiency
Join Cohere’s team to enhance the capabilities of frontier AI models by providing structured feedback on model outputs. You will assess accuracy, safety, and helpfulness while stress-testing systems to identify failure modes. This judgment-driven position requires strong analytical skills to handle ambiguous tasks and evolving guidelines.
What you'll do
- Evaluate and rank model responses for accuracy, tone, and safety
- Write clear justifications for preference and comparison judgments
- Stress-test models to uncover unsafe behavior or capability gaps
- Create high-quality prompts and exemplars for training datasets
- Design and apply grading rubrics and taxonomies consistently
- Label and correct errors in text, image, and structured data
- Participate in calibration exercises to ensure annotation consistency
- Report quality trends and performance issues to cross-functional teams
What you'll need
- One plus years in AI annotation, LLM evaluation, or research
- Experience applying detailed guidelines to ambiguous content
- Strong contextual judgment and sensitivity to nuance and tone
- Excellent written English and reading comprehension skills
- Ability to justify evaluations with clear, logical reasoning
- Comfort working with JSON, CSV, Markdown, XML, or YAML
- Strong attention to detail and consistency in high-volume tasks
- Self-motivated with good time management in a remote setting
Nice to have
- Fluency in an additional language besides English
- Familiarity with LLM behaviors like hallucination or sycophancy
- Instinct for probing models adversarially to find break points
Stack
- JSON
- CSV
- TSV
- Markdown
- XML
- YAML
How you'll work
- Fully remote within Canada
- Part-time commitment of 16 hours per week
- Independent contractor status
- Bring Your Own Device (laptop) required
- Asynchronous work in a global team
What you get
- CAD $30 per hour contract rate
- Flexible schedule as an independent contractor
- Opportunity to work on frontier AI models
Hiring process
- Initial screening by the Talent Team
- Multiple-choice assessment via email
- Comprehensive expertise-focused assessment
- Short video call with Operations Team
- Signing of Independent Contractor Agreement