视频数据标注 & 评测专家 / 负责人
Embedding VC · Remote · San Francisco Bay Area ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Video AI Data Lead
- Fully remote
- 3+ years experience
- Multimodal focus
- Quality control expert
Drive the creation of video annotation standards and evaluation frameworks for generative AI. You will define metrics for visual quality and temporal consistency while managing external labeling teams.
Collaborate closely with algorithm engineers to refine data pipelines for SFT and RLHF stages. Your work ensures high-quality training data and robust model benchmarking through systematic badcase analysis.
What you'll do
- Design video annotation tasks including captions, actions, scenes, and camera movements
- Build quality assurance systems using gold sets, sampling, and inter-annotator agreement
- Manage external vendors and crowdsourcing teams to ensure delivery consistency
- Iterate annotation guidelines based on model badcases and performance gaps
- Construct evaluation benchmarks for video generation quality and prompt alignment
- Combine manual and automated evaluation methods to track model improvements
- Coordinate with algorithm teams to align data standards with PT, SFT, and RLHF needs
What you'll need
- At least 3 years of experience in video, image, or multimodal data annotation
- Hands-on expertise in distinguishing and executing various video labeling tasks
- Practical experience with QA mechanisms like gold sets, IAA, and reviewer calibration
- Proven track record of collaborating with algorithm teams on badcase feedback loops
- Strong ability to streamline processes and communicate across different teams
Nice to have
- Experience with training data or evaluation for T2V, VLM, or multimodal models
- Understanding of SFT and RLHF, specifically the role of preference and ranking data
- Background in managing labeling suppliers, crowdsourcing, or expert teams
- Experience building annotation or evaluation workflows from scratch