AI Benchmark Engineer | Native Language Specialist - German (Austria) - Remote
LILT (Production) · Remote · Austria ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Fully remote
- Freelance contract
- Native German required
- Python + Shell
- AI benchmarking
You will build rigorous evaluation suites that test how AI models handle non-English data processing and locale-specific edge cases in terminal workflows. The work involves creating high-quality tasks in German to measure model performance without relying on English translation.
What you'll do
- Design realistic task environments using German datasets and files
- Identify failure points in AI prompting within your native language
- Write deterministic verifier scripts and reference implementations
- Calibrate task difficulty against various model tiers like Haiku and Opus
- Participate in a four-layer human quality control and audit process
What you'll need
- Native or near-native fluency in German with deep grammatical understanding
- High proficiency in English for communication and documentation
- At least one year of industry experience in software engineering
- Strong skills in Python, standard shell scripting, and data processing
- Extensive experience with Terminal/CLI-based development workflows
- Understanding of Unicode normalization and encoding robustness
- Familiarity with locale-dependent conventions and safe string operations
Stack
- Python
- Shell scripting
- Terminal-Bench
- Coding agents
How you'll work
- Independent contractor role with flexible scheduling
- Remote work from anywhere, subject to geographic restrictions
- Requires reliable availability of 15-20 hours per week
- No fixed hours, check-ins, or micromanagement
- Contractors manage their own tax obligations
What you get
- Competitive rates with prompt payment processing
- Access to diverse AI and language technology projects
- Opportunity to join a global community of experts
- Flexible workload suitable for supplemental income
Hiring process
- Submit application with an updated CV in English
- Complete a GenAI assessment to evaluate technical skills
- Finalize onboarding and profile setup in the system