AI Benchmark Engineer | Native Language Specialist - Japanese - Remote
LILT (Production) · Remote · Japan (Remote) ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Native Japanese required
- Freelance / Contract
- Fully remote
- Python + Shell
- AI Benchmarking
You will create high-quality evaluation tasks that test how AI models handle non-English data processing, encoding edge cases, and complex locale issues. The work involves building realistic task environments using Japanese datasets and writing deterministic verifier scripts to ensure benchmark integrity.
What you'll do
- Design realistic task environments using Japanese datasets and files
- Identify failure points where AI struggles with native language prompts
- Write reliable reference implementations and deterministic verifier scripts
- Analyze execution logs to calibrate task difficulty levels
- Participate in a four-layer human quality control and audit process
What you'll need
- Native or near-native fluency in Japanese with deep grammatical knowledge
- High proficiency in English for communication and documentation
- At least one year of experience in software or prompt engineering
- Strong skills in Python, standard shell scripting, and data processing
- Extensive experience with Terminal/CLI-based development workflows
- Understanding of Unicode normalization and encoding robustness
- Familiarity with locale-dependent conventions like collation and dates
Nice to have
- Background from leading technology companies or top-tier universities
- Experience with bidirectional text handling or font fallbacks
- Working familiarity with coding agents
Stack
- Python
- Shell scripting
- Terminal-Bench
- Coding Agents
How you'll work
- Independent contractor role with flexible scheduling
- No fixed hours, check-ins, or micromanagement
- Requires minimum 2 hours per day or 15-20 hours per week
- Not suitable as a primary income source due to fluctuating demand
- Contractors are responsible for their own tax obligations
What you get
- Competitive rates with prompt payment processing
- Access to diverse AI and language technology projects
- Opportunity to join a global community of experts
- Flexible work environment allowing you to set your own schedule
Hiring process
- Submit application with an updated CV in English
- Complete a GenAI assessment to evaluate technical skills
- Finalize onboarding and profile setup in the system