AI Agent Evaluator
Anyone AI · Remote · Argentina - Fully Remote · Uruguay · Chile · Poland · Italy · Czech Republic · Estonia · Ecuador - Fully Remote · Portugal · Mexico - Fully Remote · Paraguay · Colombia - Fully Remote · France · Spain · Peru - Fully Remote · Brazil ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Fully remote
- Part-time contract
- Up to 3 months
- AI evaluation
- Multiple languages
Anyone AI seeks experts to assess how well AI assistants understand user context and deliver helpful responses. You will engage in natural, multi-turn conversations with two AI versions, leveraging information from your connected digital services.
Your task is to compare the effectiveness of each assistant in using relevant data and responding to your needs. This feedback helps improve frontier AI models by ensuring they provide trustworthy and safe interactions.
What you'll do
- Write prompts based on your personal questions, interests, or needs
- Engage in natural, multi-turn conversations with two AI assistants
- Incorporate questions that utilize data from connected services
- Compare how effectively each assistant uses available context
- Assess responses for helpfulness, natural flow, trust, and safety
- Evaluate how assistants handle feedback and corrections
- Provide clear explanations to support your evaluations
What you'll need
- Fluency in one of the supported languages and locales
- Strong reading comprehension and clear written communication
- Attention to detail and sound judgment in comparisons
- Willingness to use personal context and connected services
- Recent activity across a qualifying combination of digital services
- Successful completion of eligibility verification process
Nice to have
- Experience using AI assistants for daily tasks or research
- Background in AI evaluation, user research, or QA
- Ability to spot subtle differences in personalization quality
How you'll work
- Fully remote position
- Independent contractor status
- Part-time engagement
- Project duration up to three months