AI Benchmark Engineer | Native Language Specialist - Chinese Mandarin - Remote
LILT (Production) · Remote · China (Remote) · Taiwan (Remote) ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Fully remote
- Freelance contract
- Native Mandarin required
- Python + Shell
- Flexible schedule
Join LILT as an AI Benchmark Engineer to build rigorous evaluation suites for large language models. You will create high-quality tasks in Mandarin that test model robustness in non-English terminal environments, focusing on real-world software challenges without relying on English translations.
What you'll do
- Design realistic task environments using datasets and files in Mandarin
- Identify failure points where AI struggles with native language prompts
- Write deterministic verifier scripts and reference implementations
- Calibrate task difficulty by analyzing execution logs against model tiers
- Participate in a four-layer human quality control and audit process
What you'll need
- Native or near-native fluency in Chinese Mandarin
- High proficiency in English for communication and documentation
- At least one year of industry experience in software engineering
- Strong skills in Python, standard shell scripting, and data processing
- Extensive experience with Terminal/CLI-based development workflows
- Deep understanding of Unicode normalization and encoding robustness
- Familiarity with locale-dependent conventions and text I/O operations
Nice to have
- Background from leading technology companies or top-tier universities
- Working familiarity with coding agents and AI evaluation tools
Stack
- Python
- Shell scripting
- Terminal-Bench
- Coding Agents
How you'll work
- Fully remote freelance position
- Flexible hours with no fixed schedule or micromanagement
- Requires reliable availability for accepted tasks
- Typical commitment of 15-20 hours per week
- Contractors manage their own tax obligations
Hiring process
- Submit application with updated CV in English
- Complete GenAI assessment to evaluate technical skills
- Finalize onboarding and profile setup in the system