AI Evaluation Rubric Auditor | $60-$85/Hour
- Salary
- $60–$85/hr
- Hiring from
- Argentina
- Work type
- Remote
- Posted
623,259 remote jobs, straight from company career pages
100% free · New jobs every minute
Show job descriptionHide job description
We are looking for experienced AI evaluation professionals to audit the quality and accuracy of LLM evaluation tasks.
In this role, you will review work completed by AI Evaluation Reviewers, assess their decisions, and ensure that prompts and grading rubrics are fair, accurate, and aligned with evaluation guidelines.
You will also provide actionable feedback to reviewers to improve the overall quality and consistency of AI evaluation.
Responsibilities
- Audit complete AI evaluation tasks, including prompts, grading rubrics, and model attempts.
- Assess whether identified model failures are genuinely justified.
- Verify whether the required information was available to the AI agent.
- Identify inaccurate, ambiguous, or unfair grading criteria.
- Determine whether valid model responses could incorrectly receive failing grades.
- Check rubric structure and compliance with evaluation guidelines.
- Identify tasks that cannot be reasonably corrected.
- Provide clear, constructive feedback to reviewers on their decisions.
- Maintain consistent auditing quality and availability.
Requirements
- Substantial experience writing, reviewing, or auditing LLM evaluation rubrics.
- Previous experience in AI evaluation, RLHF, model training, or AI quality assurance.
- Experience providing feedback to writers, reviewers, or evaluators.
- Strong analytical and critical-thinking abilities.
- Excellent written English and attention to detail.
- Ability to analyze lengthy AI agent transcripts and identify specific errors.
- Ability to independently verify information rather than blindly trusting reference answers.
- Consistent availability and ability to meet project expectations.
Relevant Experience
Candidates with experience in the following areas are encouraged to apply:
- LLM Evaluation and Benchmarking
- Rubric Design and Auditing
- RLHF and Human Feedback
- AI Quality Assurance
- AI Model Assessment
- Reviewer Quality Control
- Prompt and Response Evaluation
- Technical Content Review
Application Requirement: Please list the AI evaluation projects you have worked on, the organizations or platforms involved, and your specific responsibilities.
Contract Details
- Compensation: $60-$85 per hour
- Work arrangement: Fully Remote
- Schedule: Flexible, with consistent availability expected
- Payments: Weekly via Stripe or Wise
Join a team working with leading AI labs to improve the quality, reliability, and evaluation of next-generation AI systems.
About Hirely
Hirely connects qualified professionals with remote opportunities in AI research, business, software engineering, and technology. We are recruiting candidates on behalf of a hiring partner, which manages final candidate selection, onboarding, project assignments, and engagement terms.
