AI Reliability Manager

AI Reliability Manager Job Description: The Risk & Fraud Product Management team is looking for an AI Reliability Manager to help ensure the agentic AI capabilities in CLEAR, our investigative platform, are accurate, trustworthy, and continuously improving. Our customers include law enforcement, financial crimes compliance teams, corporate fraud investigators, and government agencies who use our products to make consequential decisions. The quality of what our AI produces matters enormously, and this role exists to measure that quality, defend it, and drive it upward. The AI Reliability Manager is an individual contributor who is an analytically rigorous, resourceful problem-solver who is genuinely curious about how agentic AI systems work and how to make them better. You and the reliability team will own the evaluation data and quality signals that tell us whether our AI is performing to standard, translate customer-reported issues into actionable engineering work, and serve as a trusted voice on release readiness. This is a hands-on role for someone who likes digging into messy output, finding the pattern in it, and turning that pattern into tangible improvements. You will join an established evaluation program with existing gold datasets, annotation guidelines, and scoring rubrics already in production use, and you will be responsible for extending and scaling that work as our AI capabilities expand. Key Responsibilities Evaluation & Quality Measurement Own and extend the gold datasets, annotation guidelines, and scoring rubrics used to evaluate agentic AI output for accuracy, completeness, and appropriate sourcing. Execute evaluation cycles, scoring AI responses against established guidelines, maintaining consistency across annotators, and flagging where output falls short. Analyze evaluation results to identify failure patterns, quantify their impact, and recommend where engineering and data science effort should be focused. Test and validate AI features prior to release, and provide a clear, evidence-based point of view on release readiness. Train and support additional cross functional annotator resources, including globally distributed contributors who participate in evaluation cycles but are not day-to-day experts, ensuring they apply guidelines consistently. Product Support & Issue Resolution Serve as a day-to-day product resource for the sales channel, answering questions about product behavior, capabilities, and known limitations, and keeping them current on the status of open issues. Own intake of reported issues across the CLEAR application, from AI output quality concerns to general product defects; reproduce issues, determine root cause category, and document them precisely. Open and manage engineering and labs tickets, and drive them to resolution alongside product, engineering, and data science partners. Track recurring defect themes over time and surface them to product leadership as systemic issues rather than one-off tickets. Cross-Functional Partnership Partner with product management, engineering, applied research, and go-to-market teams to translate quality findings into roadmap and prioritization decisions. Build an understanding of customer pain points and goals, and help identify new ways agentic AI can be applied to investigative workflows. Required Qualifications Three or more years, or equivalent experience, in work requiring careful judgment about quality, such as research, analysis, editorial, audit, quality assurance, or investigative work Experience applying consistent standards to open-ended work where reasonable people can disagree about what "good" looks like Hands-on use of generative AI tools, with enough curiosity to have noticed how they fail, including confident answers that aren't supported and sources that don't say what the AI claims Strong written communication, including explaining technical findings to non-technical audiences Preferred Qualifications: Experience in one of our primary customer segments: law enforcement or criminal investigations, AML/BSA or financial crimes compliance, corporate fraud or corporate security, or government program integrity. Working understanding of how retrieval-based AI systems generate and cite sources, and of common failure modes including hallucination, citation mismatch, and incomplete retrieval. #LI-DS4 What’s in it For You? Hybrid Work Model: We’ve adopted a flexible hybrid working environment for our office-based roles while delivering a seamless experience that is digitally and physically connected. Flexibility & Work-Life Balance: Flex My Way is a set of supportive workplace policies designed to help manage personal and professional responsibilities, whether caring for family, giving back to the community, or finding time to refresh and reset. This builds upon our flexible work arrangements, including work from anywhere for up to 8 weeks per year, empowering employees to achieve a better work-life balance. Career Development and Growth: By fostering a culture of continuous learning and skill development, we prepare our talent to tackle tomorrow’s challenges and deliver real-world solutions. Our Grow My Way programming and skills-first approach ensures you have the tools and knowledge to grow, lead, and thrive in an AI-enabled future. Industry Competitive Benefits: We offer comprehensive benefit plans to include flexible vacation, two company-wide Mental Health Days off, access to the Headspace app, retirement savings, tuition reimbursement, employee incentive programs, and resources for mental, physical, and financial wellbeing. Culture: Globally recognized, award-winning reputation for inclusion and belonging, flexibility, work-life balance, and more. We live by our values: Obsess over our Customers, Compete to Win, Challenge (Y)our Thinking, Act Fast / Learn Fast, and Stronger Together. Social Impact: Make an impact in your community with our Social Impact Institute. We offer employees two paid volunteer days off annually and opportunities to get involved with pro-bono consulting projects and Environmental, Social, and Governance (ESG) initiatives. Making a Real-World Impact: We are one of the few companies globally that helps its customers pursue justice, truth, and transparency. Together, with the professionals and institutions we serve, we help uphold the rule of law, turn the wheels of commerce, catch bad actors, report the facts, and provide trusted, unbiased information to people all over the world. In the United States, Thomson Reuters offers a comprehensive benefits package to our employees. Our benefit package includes market competitive health, dental, vision, disability, and life insurance programs, as well as a competitive 401k plan with company match. In addition, Thomson Reuters offers market leading work life benefits with competitive vacation, sick and safe paid time off, paid holidays (including two company mental health days off), parental leave, sabbatical leave. These benefits meet or exceeds the requirements of paid time off in accordance with any applicable state or municipal laws. Finally, Thomson Reuters offers the following additional benefits: optional hospital, accident and sickness insurance paid 100% by the employee; optional life and AD&D insurance paid 100% by the employee; Flexible Spending and Health Savings Accounts; fitness reimbursement; access to Employee Assistance Program; Group Legal Identity Theft Protection benefit paid 100% by employee; access to 529 Plan; commuter benefits; Adoption & Surrogacy Assistance; Tuition Reimbursement; and access to Employee Stock Purchase Plan. Thomson Reuters complies with local laws that require upfront disclosure of the expected pay range for a position. The base compensation range varies across locations. For any eligible US locations, unless otherwise noted, the base compensation range for this role is $91,000 USD - $169,000 USD. Base pay is positioned within the range based on several factors including an individual’s knowledge, skills and experience with consideration given to internal equity. Base pay is one part of a comprehensive Total Reward program which also includes flexible and supportive benefits and other wellbeing programs. This role may also be eligible for an Annual Bonus based on a combination of enterprise and individual performance. About Us Thomson Reuters informs the way forward by bringing together the trusted content and technology that people and organizations need to make the right decisions. We serve professionals across legal, tax, accounting, compliance, government, and media. Our products combine highly specialized software and insights to empower professionals with the data, intelligence, and solutions needed to make informed decisions, and to help institutions in their pursuit of justice, truth, and transparency. Reuters, part of Thomson Reuters, is a world leading provider of trusted journalism and news. We are powered by the talents of 26,000 employees across more than 70 countries, where everyone has a chance to contribute and grow professionally in flexible work environments. At a time when objectivity, accuracy, fairness, and transparency are under attack, we consider it our duty to pursue them. Sound exciting? Join us and help shape the industries that move society forward. As a global business, we rely on the unique backgrounds, perspectives, and experiences of all employees to deliver on our business goals. To ensure we can do that, we seek talented, qualified employees in all our operations around the world regardless of race, color, sex/gender, including pregnancy, gender identity and expression, national origin, religion, sexual orientation, disability, age, marital status, citizen status, veteran status, or any other protected classification under applicable law. Thomson Reuters is proud to be an Equal Employment Opportunity Employer providing a drug-free workplace. Thomson Reuters makes reasonable accommodations for applicants with disabilities, including veterans with disabilities, and for sincerely held religious beliefs in accordance with applicable law. If you reside in the United States and require an accommodation in the recruiting process, you may contact our Human Resources Department at HR.Leave-Expert@thomsonreuters.com . Disability accommodations in the recruiting process may include things like a sign language interpreter, making interview rooms accessible, providing assistive technology, or other relevant accommodations. Please note this email is not intended for general recruitment questions and we will promptly respond to inquiries regarding accommodations. More information on requesting an accommodation here. Learn more on how to protect yourself from fraudulent job postings here. More information about Thomson Reuters can be found on thomsonreuters.com