Remote | AI Evaluation Guidelines & Rubric Specialist — $40–$60/hour

We are sharing a specialised full-time consulting opportunity for US-based linguists, instructional designers, and technical writers experienced in developing clear evaluation guidelines, structured rubrics, and human-rating instructions for generative AI programmes. This role supports a high-impact generative AI initiative focused on translating complex and potentially ambiguous programme requirements into precise, practical guidance for human evaluators. Selected professionals will develop rater-ready instructions across domains such as finance, retail, insurance, legal, and sports while resolving contradictions, defining edge cases, and improving consistency throughout evaluation workflows. Key Responsibilities Rater Guideline Development Translate programme requirements into clear, structured, and actionable instructions for human evaluators Develop guidelines that can be applied consistently across standard scenarios and complex edge cases Define terminology, rating criteria, decision rules, exceptions, and escalation pathways Ensure instructions are accessible to raters while preserving necessary domain-specific precision Rubric & Evaluation Framework Design Design detailed scoring rubrics for evaluating generative AI outputs Establish measurable criteria covering correctness, relevance, reasoning quality, completeness, and instruction adherence Create examples and counterexamples illustrating different performance levels Align evaluation frameworks with programme objectives and quality standards Ambiguity & Consistency Review Review draft specifications for ambiguity, contradiction, missing information, and inconsistent terminology Identify instructions that may lead to conflicting interpretations across raters Revise guideline sets until they can be applied reliably with minimal escalation Document concrete before-and-after improvements to written requirements and evaluation instructions Cross-Domain Instructional Translation Convert specifications from finance, retail, insurance, legal, sports, and other specialist domains into rater-ready guidance Collaborate with subject matter experts to understand domain-specific terminology and professional judgment Preserve important technical nuance while making instructions clear to non-specialist evaluators Maintain consistent structure and quality across multiple domain-specific guideline sets Ideal Profile Strong candidates may have: At least 3 years of professional experience in linguistics, instructional design, technical writing, content design, or a closely related field Direct experience developing or refining guidelines and rubrics for human evaluators in generative AI, RLHF, or model-assessment programmes Demonstrated ability to resolve ambiguity and contradiction in complex written specifications Experience translating specialist requirements into clear and practical instructions Ability to work effectively across multiple subject-matter domains A portfolio or concrete examples showing measurable improvements to guidelines, rubrics, or instructional materials Demonstrable professional growth and increasing responsibility Reliable availability for at least 35 hours per week during weekdays Educational Background A degree in linguistics, instructional design, education, communications, technical writing, language studies, or a related field is highly relevant Graduate-level education in applied linguistics, learning design, human-computer interaction, or information design may be helpful Equivalent professional experience in AI evaluation, technical documentation, or guideline development may also be considered Training in assessment design, taxonomy development, content strategy, or quality assurance may be valuable Nice to Have Experience supporting large language model evaluation, reinforcement learning from human feedback, or AI training-data programmes Familiarity with annotation platforms, human-feedback workflows, and rater calibration processes Experience developing domain-specific guidance for finance, insurance, retail, legal, sports, or comparable fields Knowledge of controlled language, information architecture, taxonomy design, or content governance Experience conducting guideline usability tests or analysing inter-rater consistency Familiarity with version control, documentation systems, and structured authoring tools Previous collaboration with researchers, programme managers, engineers, and subject matter experts Why This Opportunity Apply linguistic and instructional-design expertise to an advanced generative AI initiative Influence the clarity and reliability of human evaluation processes Develop guidelines used across a wide range of professional subject-matter domains Solve complex problems involving ambiguity, edge cases, and evaluation consistency Join a full-time remote engagement with competitive hourly compensation Contract Details Full-time W-2 contingent employment arrangement Fully remote role available to candidates based in the United States Expected commitment of at least 35 hours per week during weekdays Competitive rates between $40–$60 per hour depending on expertise and project scope Direct experience developing rater guidelines or rubrics for generative AI or RLHF programmes is required Applicants should be prepared to provide concrete examples of guideline or specification improvements Immediate availability is preferred Work may include onboarding, calibration, documentation review, and ongoing guideline refinement Project scope and duration may be adjusted according to programme requirements and performance About the Platform This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams. By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy .