Applied AI Evaluation & Product Researcher (Contract)

Job title

Applied AI Evaluation & Product Researcher (12 Month Contract)

Department

Technology & Data

Reports to

Director, Applied Research

Salary range

$95,000 - $110,000

Job type

Full-Time, Contract, Hybrid working 1-2 days per week

Location

Toronto, Ontario

Posting period

Tuesday, November 17, 2026

The opportunity

Are you motivated by the chance to help ensure all young people in Canada can access mental health support and Feel Out Loud? Kids Help Phone (KHP) is Canada’s Unlock The Hope Youth Charity. We’re the first and Canada's only national 24/7, free, confidential, and multilingual e-mental health service for youth and a global leader in innovation for youth and their well-being.

KHP is looking for an Applied AI Evaluation & Product Researcher to lead product evaluation and user-centred research for its youth-facing generative AI product. This first-of-its-kind experience will offer low-intensity support through AI, incorporating human oversight informed by KHP’s 37+ years of supervising quality mental health support at scale.

Working with Data and Technology, Innovation, Applied Research, Clinical Services, and other internal teams, the researcher will develop and implement an evaluation strategy that defines success, examines user experience and risk, and translates findings into practical recommendations. They will also support coordination across teams and help develop clinical governance mechanisms for the initiative. It’ll take courage, collaboration, innovation, resilience and trust. If you’re interested in transforming e-mental health solutions with KHP, you can explore this opportunity today!

The responsibilities

Evaluation Strategy and Measurement 

• Develop and implement a comprehensive evaluation strategy for the system, from early testing through launch and post-launch monitoring
• Create and maintain an evaluation matrix linking objectives, user scenarios, outcomes, indicators, data sources, timing, and decision criteria
• Define measures of success for service users including user experience, adoption, trust, safety, equity, operational performance, and impact
• Design evaluation activities and coordinate the collection, analysis, and interpretation of appropriate data
• Establish processes for ongoing monitoring, learning, and continuous improvement, and communicate findings to project teams and leadership
• Coordinate evaluation activities across internal workstreams and ensure that key evaluation decisions, findings, risks, dependencies, and resulting actions are documented and communicated to the appropriate teams

User Experience Research

• Support user research to understand user needs, behaviours, expectations, barriers, and experiences across the end-to-end service journey
• Design and conduct post-interaction surveys, usability testing, journey mapping, co-design activities, and other appropriate research methods 
• Ensure research activities are inclusive, trauma-informed, accessible, and appropriate for the intended users and service context
• Translate findings into clear recommendations for product design, user flows, service pathways, content, and implementation

Safety, Risk and Responsible AI 

• Co-develop evaluation criteria, testing approaches, monitoring requirements, and escalation pathways for identified risks (e.g., user, clinical, operational, ethical, privacy, accessibility, and equity-related risks)
• Assess unintended outcomes, inappropriate or unclear recommendations, potential bias, and other issues that may affect user safety or access to support
• Contribute research evidence to responsible AI, privacy, and data governance processes and decisions
• Collaborate with designated clinical leaders and subject-matter experts to ensure evaluation criteria reflect relevant clinical standards, service practices, and safety considerations

Implementation and Launch Readiness 

• Work with internal teams to define launch-readiness criteria, decision points, evidence requirements, and timelines, and provide evaluation findings to support decisions by accountable product, operational, and clinical leaders
• Identify user-experience issues that may affect implementation and recommend strategies for improvement
• Support iterative testing throughout design, development and implementation and evaluate whether changes address identified user needs and risks

Reporting and Knowledge Translation 

• Synthesize complex research and evaluation findings into practical recommendations, reports, presentations, briefings, and decision materials
• Conduct literature reviews, environmental scans, and evidence syntheses relevant to Generative AI, digital services, user experience, safety, and implementation
• Communicate research limitations, emerging risks, and evidence gaps clearly to technical and non-technical audiences

The qualifications

• PhD degree (or Master’s degree with equivalent experience) in Human-Computer Interaction, User Experience Research, Psychology, Social Sciences, Evaluation, Implementation Science, Health Informatics, or a related field
• Minimum 5 years of experience in generative AI systems evaluation, UX research, product research and evaluation, applied research, evaluation, digital service design, or a related area
• Strong knowledge of responsible AI, AI safety, risk assessment, algorithmic evaluation, or technology governance
• Strong experience with qualitative and quantitative research methods, including surveys, usability testing, and analysis
• Experience developing evaluation frameworks, measurement strategies, success indicators, logic models, or evaluation matrices
• Experience translating research findings into actionable product, service, or implementation recommendations
• Experience collaborating with multidisciplinary teams that may include product, design, technology, data, research, and clinical partners
• Strong project coordination, facilitation, stakeholder engagement, and written and verbal communication skills
• Ability to manage multiple priorities, work through ambiguity, and communicate risks, trade-offs, and recommendations clearly

Prefererred

• Experience working in mental health, youth services, healthcare, crisis support, or another safety-sensitive environment
• Familiarity with clinical standards, privacy, accessibility, equity, and data governance considerations
• Experience supporting product pilots, implementation readiness, launch activities, and post-launch monitoring
• Familiarity with research, analytics, prototyping, or product collaboration tools

Don’t meet every qualification listed? Studies often report equity-deserving people are less likely to apply for jobs unless they meet every requirement. KHP is dedicated to building a more diverse, inclusive and authentic workplace, so if you’re excited about this role, but your experience doesn’t align with every qualification, we welcome you to apply anyway. You may have the qualifications / experience for this role or other opportunities at KHP!

The application process

To apply for this opportunity, you can tap the “Apply” button on this page and follow the on-screen instructions.

About KHP, Canada’s Unlock The Hope Youth Charity

Kids Help Phone (KHP) is Canada’s only free, national, 24/7, multilingual and confidential e-mental health service for youth to get help and Feel Out Loud. Since 1989, KHP has been the empowering, trusted changemaker giving young people’s feelings, big and small, a non-judgmental place to go through personalized well-being solutions. A made-in-Canada, global leader, KHP continually evolves by blending data, innovation and technology with real human support and clinical knowledge to unlock the hope young people need to thrive in their world. Explore help for all the feelings at KidsHelpPhone.ca.

Inclusion, Diversity, Equity and Accessibility (IDEA)

KHP has a role to play in effecting social change and advancing equity. This is why the practice of IDEA is critical to us. KHP has a long history of celebrating and advancing IDEA through our tailored programs, services, policies, initiatives and task forces with contributions and leadership from teams across the organization. We will not waver on these commitments and will continue leading the way in making the workplace, and the world young people live in, more inclusive, diverse, equitable and accessible.

We believe it’s critical that our services empower everyone who interacts with us to bring their authentic selves to an environment of mutual respect. An open and diverse community can foster the inclusion and amplification of voices that have been, and continue to be, underrepresented.

As such, we welcome applicants from equity-deserving communities (e.g. Indigenous Peoples, racialized persons, persons with disabilities, women, 2SLGBTQ+ individuals, etc.). We strongly believe that alleviating the underrepresentation of equity-deserving groups will help create a stronger KHP and help us better support the needs of youth from coast to coast to coast.

To learn more about how we are committed to advancing IDEA, you can explore KHP’s IDEA 2023–2027 Strategy and Action Plan.

Accommodations

KHP welcomes applications from people living with disabilities. Reasonable accommodations are available upon request during any phase of the application / hiring process. If you’re contacted about an opportunity, you can reply and let us know if you require accommodations at any time.

Compensation and benefits

Salaries at KHP are determined by candidate experience, qualifications and specific skill sets, as well as market and operational considerations. Potential additional benefits at KHP include:

• Hybrid work model
• Comprehensive Employee Benefits Program
• Emergency Travel Assistance
• CAAT Pension Plan
• Wellness programs
• Recognition awards
• Employee and Family Assistance Program (EFAP)

Candidate verification

The successful candidate for this opportunity will be asked to complete a background screen, which may include criminal, credit, employment and educational checks. All offers of employment are conditional on the receipt of satisfactory results of any applicable check.

Privacy

KHP cares about your privacy! We’re required to save select information about opportunities, applications and interviews for set periods of time. To learn more, you can explore KHP’s Privacy Policy or contact us using our general form.

Message to agencies

KHP doesn’t accept unsolicited phone calls, referrals and / or resumes from any source other than directly from candidates. KHP won’t consider unsolicited phone calls, referrals and / or resumes from vendors including and without limitation, search firms, staffing agencies, fee-based referral services and recruiting agencies. The submission of unsolicited referrals and / or resumes by anyone other than a candidate directly to KHP is deemed gratuitous. KHP won’t be obligated or bound in any way to pay a referral or any other fee if a person referred to us from a source other than a direct candidate is subsequently hired.