AI Evals Engineer — Evaluation Datasets & Ground Truth
Prophetic Technologies Inc · Remote · US (Remote) · Portland, OR ·
COMPENSATION
€•• – ••k
Listed on the employer's job board (Ashby)
- US work authorization required
- Remote or Hybrid (Portland)
- Python + SQL
- LLM evaluation focus
- No visa sponsorship
You will own the ground truth for Prophetic’s AI-native real estate platform, creating versioned datasets that measure pipeline performance. The role involves defining correctness rubrics, sourcing data via production sampling or synthetic oracles, and validating labels to support engineering iterations.
What you'll do
- Decide what metrics matter and decompose systems into evaluable modules
- Define written rubrics and label schemas for correctness
- Source data via production sampling, human labeling, or synthetic oracles
- Manage labeling vendors and ensure inter-annotator agreement
- Version datasets and guard against contamination or leakage
- Calibrate automated graders against human gold sets
- Report precision, recall, and confidence intervals to engineering teams
What you'll need
- Experience building or evaluating production ML or LLM systems
- Proven track record creating validation datasets with defined correctness
- Fluency in ML validation fundamentals like stratified sampling and calibration
- Strong Python and SQL skills for data pulling and reshaping
- Hands-on experience with prompting, structured outputs, and agent harnesses
- Ability to judge reliability of LLM-as-judge approaches
- Capacity to translate system design into label schemas independently
Nice to have
- Experience running end-to-end human labeling programs
- Familiarity with eval tooling and data labeling platforms
- Experience using frontier models as distillation or oracle sources
Stack
- Python
- SQL
- LLMs
How you'll work
- Remote, hybrid, or in-office options available
- Headquarters located in Portland, OR
- Must be authorized to work in the United States
- No visa sponsorship now or in the future
What you get
- 100% medical, dental, and vision for employee
- 30% coverage for dependents
- Competitive salary and early-stage equity
- Unlimited PTO
- Hybrid or remote stipend
- Budget for intra-office travel
- 2–3 annual team meetups