Senior Software Engineer – Open Source & SWE-Bench Evaluation
Anyone AI · Remote · Argentina - Fully Remote · Uruguay · Chile · Ecuador - Fully Remote · Portugal · Mexico - Fully Remote · Colombia - Fully Remote · Spain · Brazil ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- Senior level
- Part-time contract
- Fully remote
- Open source focus
- Code evaluation
Join Anyone AI as a consultant to assess real-world software engineering tasks derived from open-source repositories. You will evaluate whether coding challenges based on GitHub issues are clear, solvable, and representative of professional engineering work.
The position focuses on quality assurance for technical benchmarks, requiring deep expertise in code review, testing strategies, and environment configuration. It is ideal for engineers who enjoy analyzing complex codebases and providing detailed technical feedback.
What you'll do
- Review coding tasks derived from real GitHub issues and pull requests
- Assess clarity and completeness of problem statements and success criteria
- Evaluate unit tests for correctness, coverage, and robustness
- Identify flaky tests, missing dependencies, and version conflicts
- Determine if tasks are reliably reproducible across different environments
- Judge the real-world difficulty and complexity of each engineering task
- Provide recommendations on accepting, improving, or excluding tasks
What you'll need
- At least three years of professional software engineering experience
- Strong experience working with large, multi-file codebases
- Proven track record in reviewing pull requests and debugging issues
- Deep understanding of unit testing principles and test coverage
- Ability to validate tests without overly restricting implementation
- Experience with dependency management and environment reproducibility
- Strong grasp of Git and GitHub-based development workflows
- Skill in analyzing complex technical problems and writing clear feedback
Nice to have
- Contributions to or maintenance of open-source projects
- Familiarity with SWE-Bench or similar coding benchmarks
- Experience with major Python projects like Django or Flask
- Knowledge of Docker, CI/CD, pip, conda, or dependency pinning
- Understanding of test fixtures, isolation, or property-based testing
- Background in designing technical assessments or coding challenges
- Experience with AI/ML evaluation, data curation, or RLHF
How you'll work
- Fully remote work arrangement
- Part-time engagement
- Project-based consulting contract