Software Engineering Evaluator (contract, Remote)

About the role

Key Responsibilities

• Design task-specific grading criteria for feature implementations, bug fixes, refactors, tests, and code reviews.
• Evaluate AI-generated code for correctness, including failures that pass the code's own tests.
• Assess edge-case handling, security, performance, and readability in coding deliverables.
• Write clear, evidence-based justifications for every evaluation score.
• Apply consistent judgment across software engineering evaluation tasks.
• Review code that may appear correct and run without failing while identifying subtle defects.
• Reason through ambiguity and distinguish clearly between known facts and inferences during evaluations.
• Iterate on evaluation work based on feedback from senior reviewers.

Required Qualifications

• 2–3 years of professional software engineering experience shipping production code.
• Strong command of at least one of Python, JavaScript/TypeScript, Java, Go, C++, C#, or a comparable language.
• Solid grounding in data structures, algorithms, testing, debugging, and version control.
• Experience reviewing other engineers' code.
• Excellent written English.
• Ability to follow complex instructions, reason through ambiguity, and spot subtle errors in code that looks correct and runs without failing.
• Ability to clearly separate what you know from what you are inferring.
• Ability to work independently, learn new tools quickly, and take feedback well.
• Based in and working from the US, Canada, or UK.
• Available for 10+ hours per week for the full 12 weeks.

Benefits and Perks

• $50–$125 per hour, paid weekly via Stripe or Wise.
• Remote work on your own schedule within the 10+ hour weekly engagement.
• Initial 12-week contract with the possibility that projects may be extended based on needs and performance.
• Opportunity to evaluate AI systems for leading AI research organizations.
• Reasonable accommodations available upon request.

Skills

• Python
• JavaScript/TypeScript
• Java
• Go
• C++
• C#
• Data Structures
• Algorithms
• Testing
• Debugging
• Version Control
• Code Review
• Production Code
• Grading Criteria Design
• Evaluating AI-Generated Code
• Correctness Evaluation
• Edge-Case Handling
• Security
• Performance
• Readability
• Backend Systems
• Distributed Systems
• Infrastructure
• Open-Source Contributions
• Competitive Programming
• Evaluating AI Outputs
• Grading Technical Work
• Stripe
• Wise