Senior Product Software Engineer - AI Quality Engineering

We are seeking an experienced Senior AI Quality Engineer to own and implement quality strategy, evaluation systems, and reliability practices for AI-powered product experiences in Wolters Kluwer Tax and Accounting. In this role, you will architect, build, and maintain evaluation systems that measure retrieval accuracy, citation correctness, response quality, safety, performance, and overall alignment with customer intent. You will work hands-on across test automation, AI evaluation, APIs, data pipelines, observability, and release readiness while partnering with software engineers, product managers, UX, and stakeholders to ensure our generative AI systems are trustworthy, observable, secure, and ready for production use in professional tax and accounting workflows.

This role goes beyond validating that features function as designed. You will own the practical quality engineering implementation for non-deterministic AI systems: defining how quality is measured, building automated evaluation and regression capabilities, implementing quality gates in delivery pipelines, driving release-readiness decisions, mentoring team members in AI quality practices, and establishing the standards customers can trust.

Key Responsibilities:

• Own and implement the quality engineering approach for enterprise-scale generative AI applications, including RAG pipelines, LLM orchestration, agentic workflows, APIs, ingestion pipelines, and AI-driven user experiences.
• Architect and implement evaluation harnesses that measure retrieval accuracy, citation correctness, answer relevance, groundedness, hallucination rate, safety, latency, cost, and end-to-end system behavior.
• Design automated regression suites that detect quality drift across prompts, embeddings, chunking strategies, model versions, retrieval configuration, orchestration logic, and application workflows.
• Partner with developers to validate AI system architecture, service integrations, API behavior, data ingestion, observability, resilience, security controls, and production readiness.
• Lead root-cause analysis for AI quality failures using logs, traces, evaluation outputs, feedback signals, customer scenarios, and domain-specific acceptance criteria.
• Define, implement, and maintain measurable quality gates for AI releases, and advise product and engineering leaders on risk, readiness, and tradeoffs across accuracy, reliability, latency, cost, and user trust.
• Develop test automation in Python and related frameworks for APIs, services, chat workflows, data pipelines, and AI evaluation workflows within CI/CD pipelines.
• Build and maintain observability practices and dashboards that track model behavior, retrieval performance, operational health, quality trends, and customer-impacting failure modes.
• Contribute to responsible AI practices including bias and fairness checks, content safety validation, prompt injection testing, data privacy validation, and compliance-oriented evidence collection.
• Collaborate with product managers, UX, subject matter experts, and stakeholders to translate customer workflows and business requirements into testable AI quality criteria.
• Mentor engineers and quality team members in AI evaluation methods, non-deterministic testing strategies, automation design, and production-quality engineering practices.
• Stay current with emerging AI quality, LLM evaluation, RAG assessment, and LLMOps practices through research, experimentation, and community engagement.

Key Requirements:

• 5+ years of experience in quality engineering, SDET, software engineering, test automation, reliability engineering, or equivalent engineering roles.
• 2+ years of hands-on experience testing, evaluating, or validating AI/ML, LLM, RAG, or other non-deterministic systems in production or production-like environments.

• Strong proficiency with Python test automation frameworks such as Pytest, unittest, or equivalent, including experience building reusable test harnesses and regression suites.
• Experience validating REST APIs, microservices, service integrations, data ingestion pipelines, and end-to-end web or chat-based workflows.
• Practical understanding of RAG systems, vector search, embeddings, prompt behavior, model evaluation, groundedness, citation quality, and AI failure modes.
• Experience defining quality metrics and success criteria where outputs are probabilistic rather than strictly deterministic.
• Knowledge of CI/CD pipelines, Git-based workflows, automated regression testing, release gates, and modern engineering practices.
• Experience with observability, logging, tracing, dashboards, and metrics used to monitor production behavior and diagnose system failures.
• Ability to navigate ambiguity, decompose complex AI behaviors into testable risks, and communicate findings clearly to technical and non-technical stakeholders.
• Strong collaboration skills in Agile environments, with demonstrated ability to influence engineering quality practices across product, design, development, and domain expert teams.
• Commitment to responsible AI quality practices, including trust, transparency, security, privacy, fairness, and customer impact.

Desired Qualifications:

• Experience with LLM observability, tracing, and evaluation platforms such as LangSmith, Langfuse, TruLens, Arize/Phoenix, or custom evaluation and monitoring tooling.
• Experience testing RAG pipelines using vector databases or search platforms such as Azure AI Search, Pinecone, Weaviate, Elasticsearch, or similar technologies.
• Experience with LLM application frameworks such as LangChain, LlamaIndex, Semantic Kernel, or custom orchestration approaches.
• Experience with Azure OpenAI Service, Azure AI Studio, Azure cloud services, or enterprise AI deployments.
• Experience with containerized environments and cloud-native delivery practices using Docker, Kubernetes, and related tooling.
• Experience with performance, load, or resiliency testing tools such as Locust, k6, JMeter, or equivalent.
• Experience with production system observability tooling such as Azure Application Insights, Grafana, Prometheus, Datadog, OpenTelemetry, or equivalent.
• Knowledge of LLMOps, model monitoring, A/B testing, human-in-the-loop evaluation, or production feedback-loop design.
• Experience validating AI applications that handle sensitive, regulated, or professional-domain data.

• Background in tax, accounting, legal, financial, or other domains where answer accuracy and explainability are critical.

Our Interview Practices

To maintain a fair and genuine hiring process, we kindly ask that all candidates participate in interviews without the assistance of AI tools or external prompts. Our interview process is designed to assess your individual skills, experiences, and communication style. We value authenticity and want to ensure we’re getting to know you—not a digital assistant. To help maintain this integrity, we ask to remove virtual backgrounds and include in-person interviews in our hiring process. Please note that use of AI-generated responses or third-party support during interviews will be grounds for disqualification from the recruitment process.

Applicants may be required to appear onsite at a Wolters Kluwer office as part of the recruitment process.

Compensation:

$83,400.00 - $145,650.00 USD

This role is eligible for Bonus.

Compensation range listed is based on primary location of the position. Actual base salary offer is influenced by a wide array of factors including but not limited to skills, experience and actual hiring location. Your recruiter can share more information about the specific offer for the job location during the hiring process.

Additional Information:

Wolters Kluwer offers a wide variety of competitive benefits and programs to help meet your needs and balance your work and personal life, including but not limited to: Medical, Dental, & Vision Plans, 401(k), FSA/HSA, Commuter Benefits, Tuition Assistance Plan, Vacation and Sick Time, and Paid Parental Leave. Full details of our benefits are available upon request.

Be the Difference

If making a difference matters to you, then you matter to us.

Join us, at Wolters Kluwer, and be part of a dynamic global technology company that makes a difference every day.

We’re innovators with impact. We provide expert software and information solutions that the world’s leading professionals rely on, in the moments that matter most. You’ll make a real difference in the lives of millions of people. Together with our customers, we help to advance sustainability, health, justice, prosperity, and commerce around the world.

You can thrive at Wolters Kluwer, where diversity is core to our collective strength and high performance. Be your unique self, share your creative ideas, do your best work, and take time to grow in our caring and inclusive culture where you can belong.

About Wolters Kluwer

Wolters Kluwer reported 2025 annual revenues of €6.1 billion. The group serves customers in over 180 countries, maintains operations in over 40 countries, and employs more than 21,000 people worldwide. ​

​

Our customers work in industries that impact the lives of millions of people every single day. Our mission is to empower our professional customers with the AI-powered solutions they need to make critical decisions, achieve successful outcomes, and increase productivity. ​

​

We deliver trusted, AI-powered expert solutions that combine deep domain knowledge, proprietary content, and advanced technology to provide expert-validated insights, automate workflows, and drive better outcomes. Today, nearly 70% of our digital revenues come from AI-powered solutions, reflecting our strategic focus on embedding AI into professional workflows and delivering measurable productivity gains. ​

​

We are committed to helping professionals navigate complexity, improve the way they work, and solve critical challenges through cloud-based, AI-powered platforms that integrate seamlessly into their workflows and ecosystems. Our solutions are designed to deliver impact when it matters most, especially in high-stakes environments where accuracy and trust are essential. ​

​

With a nearly 190-year legacy, Wolters Kluwer continues to evolve its portfolio of digital solutions and services to meet the changing needs of professionals worldwide, enabling them to work faster, make smarter decisions, and deliver better outcomes.​

​

For more information about our solutions and organization, visit www.wolterskluwer.com, or follow us on LinkedIn, Instagram and Facebook.

Tips & Tricks - Resumes

Wolters Kluwer has a dedicated team of experienced talent acquisition professionals who are ready to hear what makes you tick and share how Wolters Kluwer can help you achieve your goals and ambitions. Here Rasi Fawaz shares some tips on what recruiters are looking for on a resume.

Building Diverse Teams

As a global organization, we recognize that solving a wide range of complex problems requires diverse perspectives and innovative thinking. We know that bringing our best to our customers, communities, and other key stakeholders requires a highly engaged and talented workforce – one that represents the diversity of those we serve and the communities where we live and work.

We aim to provide a welcoming environment and equitable opportunities for all employees regardless of background, nationality, race, ethnicity, gender, gender identity, age, sexual orientation, marital status, disability, or religion.

This principle is ingrained in our company values and articulated in our Code of Business Ethics.

Our Global Well-being Program

At Wolters Kluwer, you’ll feel valued for your contributions, and you’ll know that your health, safety, and well-being are important to us. Our well-being benefits provide tools, programs, and resources to help our diverse employees feel healthy, happy, safe, and prosperous.​

​

Our global Together we thrive well-being program reflects our commitment to supporting colleagues across physical, mental, social, and financial well-being. That commitment has been recognized externally through Ragan’s Top Places to Work for Employee Wellbeing: Large Organization in 2024. ​

​

More recently, Wolters Kluwer was recognized by Forbes as one of America’s Best Large Employers 2026 and America’s Best Employers for Engineers 2026. In 2025, Wolters Kluwer was also recognized by Forbes as one of America’s Best Employers for Women.

Equal Opportunity Employer

To ensure we continue to drive innovation that enables us to develop products and services to best serve our customers, we cultivate a workplace culture rooted in mutual respect, bringing forward insights from a wide range of backgrounds, perspectives, and experiences. We are also committed to complying with laws requiring equal opportunity in hiring, promotion, and other employment decisions.

All qualified applicants will receive consideration without regard to race, color, religion, sex (including pregnancy, gender identity, transgender status, and sexual orientation), national origin, disability, age, genetic information, veteran status, or any other characteristic protected by applicable law, and we do not tolerate discrimination on any of these bases.

GDPR

Careers Privacy and Cookies

Wolters Kluwer (“we” or “us”) wants to inform you about the ways we process your personal information. In this Privacy & Cookie Notice we explain what personal information we collect, use and disclose. You'll find details related to GDPR and other data privacy policies on our Careers Privacy and Cookies page.