AI Quality & Evaluation Specialist

We’re hiring an AI Quality & Evaluation Specialist to help improve the quality and reliability of an AI-powered platform used in real-time security operations. The platform monitors live inputs such as video, audio, location data, GPS, telemetry, and chat messages to identify potential incidents, send alerts to officers, and escalate events for operational review.

This role follows a human-in-the-loop approach, where human judgment and feedback help AI systems improve their performance and decision-making. You’ll contribute to the ongoing development of AI technology by helping ensure its outputs are accurate, appropriate, and reliable in real-world situations.

This is a great opportunity for someone who enjoys working with technology, analyzing information, solving problems, and improving processes. If you’re detail-oriented, naturally curious, and interested in helping AI systems become more effective over time, we’d love to hear from you.

Responsibilities:

• Review AI-generated alerts, messages, and actions to determine whether they are correct, appropriate, and supported by the available information.
• Identify and label incorrect AI outputs, including false positives, inaccurate location or status information, inappropriate messages, and other errors.
• Document why an AI-generated action was incorrect and provide clear, actionable feedback to help improve its accuracy and decision-making.
• Evaluate correct AI outputs to help identify successful decisions and maintain consistent quality standards.
• Identify recurring errors and patterns in AI-generated outputs and communicate findings to the appropriate teams.
• Maintain accurate records of evaluations, error classifications, and feedback using internal tools and systems.
• Apply critical thinking and sound judgment when reviewing alerts, messages, and other AI-generated actions.
• Follow established evaluation guidelines and flag unclear or unusual cases for further review when needed.
• Collaborate with operations and other internal teams to share findings and support continuous improvements to the AI platform.
• Help improve evaluation criteria, review guidelines, and workflows to make the quality review process more consistent and effective.

Requirements:

• 3+ years of experience in Operations, Quality Assurance, Data Review, Process Improvement, Customer Support, or a related field.
• Experience reviewing AI-generated outputs, evaluating AI systems, or working with AI-powered tools is a big plus.
• Experience working with US-based companies or teams is a plus.
• Experience as a Team Lead or Subject Matter Expert is a plus.
• Familiarity with AI tools such as ChatGPT and a genuine interest in understanding how AI systems work and improve.
• Strong analytical and critical thinking skills, with the ability to assess information, identify errors, and distinguish correct outputs from incorrect or misleading ones.
• Excellent attention to detail and the ability to recognize inconsistencies, false alarms, data errors, and inappropriate messages.
• Strong written English communication skills, especially when documenting errors, explaining decisions, and providing actionable feedback.
• A process improvement mindset, with the ability to identify recurring issues and suggest practical solutions.
• Naturally inquisitive and comfortable asking questions to understand how systems work and why errors occur.
• A fast learner who takes feedback well and adapts to changing guidelines and processes.
• Able to work independently while maintaining accuracy, consistency, and productivity.
• Comfortable navigating SaaS tools and learning new systems in a fast-paced environment.
• Must be available to work graveyard hours aligned with U.S. time zones.
• Reliable internet connection and dedicated work-from-home setup required.

Salary and Benefits:

• Base salary: ₱65,000 per month.
• Paid Time Off and paid holidays.
• 100% work from home.