AI Benchmark Engineer | Native Language Specialist - Portuguese (Brazil) - Remote
LILT (Production) · Remote · Brazil (Remote) ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Native Portuguese (Brazil)
- Fully remote freelance
- Python + Shell scripting
- AI benchmark creation
- Flexible schedule
Join LILT as an AI Benchmark Engineer to build rigorous evaluation suites for Terminal-Bench tasks. You will create high-quality assets in Portuguese to measure how well AI models handle multilingual environments, focusing on robustness against prompt language effects and locale-specific challenges.
This is a remote freelance opportunity suited for supplemental income, offering flexible scheduling without fixed hours or micromanagement. You will collaborate with a global community of experts to advance multilingual AI capabilities.
What you'll do
- Design realistic task environments using datasets and files in Portuguese
- Identify failure points where AI struggles with native language prompts
- Write deterministic verifier scripts and reference implementations
- Calibrate task difficulty by analyzing execution logs across model tiers
- Participate in a four-layer human quality control and audit process
What you'll need
- Native or near-native fluency in Portuguese with deep grammatical knowledge
- High proficiency in English for communication and documentation
- At least one year of industry experience in software or prompt engineering
- Strong skills in Python, standard shell scripting, and data processing
- Extensive experience with Terminal/CLI-based development workflows
- Understanding of Unicode normalization and encoding/decoding robustness
Nice to have
- Background from leading technology companies or top-tier engineering universities
- Familiarity with coding agents and their operational limits
- Knowledge of locale-dependent conventions like collation and casing
Stack
- Python
- Shell scripting
- Terminal-Bench
- Coding agents
How you'll work
- Fully remote freelance position (1099 contractor)
- Flexible hours with no fixed schedule or check-ins
- Requires reliable availability for 15-20 hours per week
- Not suitable as a primary income source or full-time job
- Contractors responsible for their own tax obligations
What you get
- Competitive rates with prompt payment processing
- Freedom to choose work volume and timing
- Access to diverse AI and language technology projects
- Opportunity to expand professional portfolio and skills
Hiring process
- Submit application with an updated CV in English
- Complete a GenAI assessment to evaluate technical skills
- Finalize onboarding and profile setup in the system