Software Engineer – AI Preference Data & Evaluation

Software Engineer – AI Preference Data & Evaluation

📍 Location: Remote – United States (Approved States Only)

🌎 Schedule: Minimum 4-hour overlap with Pacific Time (PST)

⚠️ Location Requirement

Candidates must currently reside in one of the following approved U.S. states:

Alabama, Arizona, Colorado, Florida, Idaho, Iowa, Kentucky, Michigan, Minnesota, Mississippi, Missouri, Montana, New York, North Carolina, North Dakota, Oklahoma, Pennsylvania, South Carolina, South Dakota, Texas, Virginia, Wisconsin, or Wyoming.

Applications from candidates located outside these states cannot be considered for this opportunity.

About the Role

We are looking for a highly skilled Software Engineer with strong technical judgment and an exceptional eye for quality.

This role is ideal for an engineer who has hands-on experience working with preference data, rubric creation, data evaluation, and engineering workflows, and who can clearly distinguish between a good technical output and an exceptional one.

You should be comfortable working in fast-moving and ambiguous environments, communicating technical findings clearly, and translating qualitative judgment into structured and scalable evaluation processes.

What You'll Do

• Design, refine, and iterate on rubrics and evaluation criteria for preference data.
• Review, evaluate, and label data while maintaining a high bar for quality and identifying subtle issues.
• Build and maintain pipelines and infrastructure supporting data generation, collection, and evaluation workflows.
• Analyze findings and translate technical work into concise, structured, and actionable recommendations.
• Collaborate closely with researchers and engineers to transform qualitative judgment into scalable processes.
• Contribute effectively in a fast-paced environment with a strong bias toward execution and delivery.

Required Qualifications

• 5+ years of professional software engineering experience.
• Strong proficiency in at least one of the following:
• Python, particularly for building and maintaining pipelines and infrastructure
• JavaScript / TypeScript
• Golang
• Experience contributing to or navigating large and complex codebases; experience with projects comparable in scale to VS Code is a plus.
• Hands-on experience with preference data collection and/or rubric creation.
• Strong written and verbal communication skills, with the ability to distill complex technical work into clear and actionable summaries.
• Excellent technical judgment and the ability to clearly articulate why one solution, implementation, or example is better than another.
• Ability to operate effectively in ambiguous, fast-moving engineering environments.
• Availability to maintain a minimum 4-hour overlap with Pacific Time (PST).

Interview Process

Step 1: AI Interview

Step 2: Coding Challenge

Step 3: 30-minute interview covering technical and cultural alignment

Application Requirements

Please make sure your application/profile includes:

• Full Name
• Contact Information
• GitHub Profile
• LinkedIn Profile

📍 Important Location Notice

This opportunity is currently restricted to candidates residing in:

Alabama | Arizona | Colorado | Florida | Idaho | Iowa | Kentucky | Michigan | Minnesota | Mississippi | Missouri | Montana | New York | North Carolina | North Dakota | Oklahoma | Pennsylvania | South Carolina | South Dakota | Texas | Virginia | Wisconsin | Wyoming

Candidates residing in other U.S. states or countries are not eligible for this specific opening.