AI Quality Engineer

Rakbank is seeking an experienced Assistant Manager – Automation and AI Engineer to lead quality engineering and testing across our AI, Generative AI, Conversational AI, Agentic AI, and Intelligent Automation initiatives. This hands-on role will play a critical part in ensuring AI-powered solutions are accurate, reliable, secure, explainable, and production-ready while driving automation excellence across the Digital & AI ecosystem. What You Will Do: Lead end-to-end quality engineering for Generative AI, Conversational AI, Agentic AI, and intelligent automation solutions. Define AI testing strategies, evaluation frameworks, quality standards, and release-readiness criteria. Evaluate and benchmark Large Language Models (LLMs) for accuracy, safety, relevance, latency, groundedness, and cost. Design and implement AI testing frameworks covering RAG evaluation, hallucination detection, AI safety, prompt validation, and model regression. Build and integrate automated testing solutions using Python, PyTest, Playwright, API automation, and CI/CD quality gates. Drive chatbot, virtual assistant, and copilot testing across multi-turn conversations, intent coverage, and English/Arabic user journeys. Mentor QA and automation teams while partnering with Product, Engineering, and Data Science teams to embed quality throughout the AI development lifecycle. What You Will Bring: Bachelor's Degree plus 8-10 years of testing experience, including at least 3 years supporting banking technology projects in a lead or SME capacity. Strong expertise in Generative AI, Conversational AI, Agentic AI testing, LLM evaluation, RAG frameworks, and AI quality assurance methodologies. Hands-on experience with Python, PyTest, Playwright, API automation, UiPath Test Suite, and intelligent automation testing. Knowledge of AI safety testing, prompt injection, jailbreak testing, hallucination detection, privacy controls, and guardrail validation. Experience integrating automated testing and AI evaluation into Azure DevOps, Jenkins, GitLab, and modern CI/CD pipelines. Strong analytical, stakeholder management, problem-solving, and mentoring capabilities, with knowledge of banking regulations, UAE PDPL, and digital banking environments. Preferred Skills: Experience in Generative AI, Conversational AI, Agentic AI, and intelligent automation testing. Strong expertise in LLM evaluation, benchmarking, prompt validation, model regression, and RAG evaluation frameworks. Hands-on experience in AI quality assurance, including hallucination detection, LLM-as-a-Judge frameworks, and AI safety testing. Proficiency in Python, PyTest, Playwright, API automation, and UiPath Test Suite for automation and RPA testing. Experience integrating automated testing into CI/CD pipelines using Azure DevOps, Jenkins, GitLab, and Git. Knowledge of test data management, synthetic data generation, performance testing, latency profiling, and AI observability. Strong understanding of Azure DevOps Test Plans, Jira, and modern AI concepts such as embeddings, vector databases, context windows, and orchestration frameworks. Banking domain knowledge with awareness of UAE PDPL, data privacy, security, and regulatory compliance requirements.