AI QA Automation Engineer

About Us UMO is a stealth-mode FinTech venture aiming to evolve the way people experience money by building a unified, AI-powered, yet deeply human modern money platform across fiat, crypto, and investments - subject to regulatory approvals. The platform is being designed to break down traditional barriers to money across access, assets, and experience, enabling simpler, more adaptive ways for people to interact with financial services. We are currently developing our MVP and navigating licensing requirements, with a multidisciplinary team of 100+ people representing 20+ nationalities. With our headquarters in the UAE and offices in Portugal and Ukraine, we are united behind a shared ambition and a relentless focus on serving our customers. About this role As a part of our team, you'll help us shape a platform that works for One — a business owner, an artist, a freelancer. In this role, you will hold end-to-end ownership over our test automation architecture across backend microservices and native mobile applications, while also owning the quality bar for our LLM-powered features — evaluating model behavior, catching hallucinations and regressions, and stress-testing our AI systems before they ever reach a customer. We provide the space for you to operate with deep technical autonomy, collaborating directly with engineering, product, and AI/ML squads to embed quality into the build pipeline without corporate clutter or micromanagement. You'll take ownership of: Backend & Mobile Automation Architecture: Build, extend, and maintain scalable automated test suites across backend systems and native mobile apps using TypeScript, Playwright, Appium, or Cucumber. API Quality Engineering: Validate complex API layers, taking direct responsibility for deep functional, data integrity, and contract testing across REST and low-latency transport interfaces. LLM Evaluation & Testing: Design and run evaluation suites for our LLM-powered features using frameworks such as RAGAS and DeepEval, measuring groundedness, answer relevancy, faithfulness, and hallucination rate against real product scenarios. AI Red Teaming & Safety Testing: Proactively probe our LLM and agent systems for prompt injection, jailbreaks, data leakage, and unsafe or biased outputs using red-teaming frameworks such as DeepTeam and PyRIT. Pipeline Integration & Continuous Delivery: Integrate and maintain test execution — including automation and LLM eval/red-team suites — within our GitHub Actions or GitLab CI pipelines, driving continuous delivery so release trains stay smooth and reliable. Cross-Functional Product Collaboration: Work side-by-side with engineering, product, AI/ML, and DevOps squads during sprint planning, code reviews, and test design reviews to ensure optimal system testability for both traditional and AI-driven features. Lightweight MLOps/MLOPs: Support prompt/model versioning, evaluation dataset management, and monitoring of LLM outputs in production, helping the team track model and prompt performance over time. AI-Powered Testing Execution: Model the QA craft using modern AI tools to accelerate test case generation, script maintenance, and automated run analysis. About you: Engineering Experience: You bring strong commercial experience in software testing and quality engineering (5+ years), with a heavy focus on building and scaling automation frameworks. TypeScript & Framework Mastery: You feel at home writing automated tests in TypeScript, utilizing tools such as Playwright, Cucumber, or similar modern test engines. Native Mobile Testing Capability: You have practical experience with native iOS and Android test automation using Appium or native framework equivalents. API & Data Precision: You have proven experience testing APIs (REST is essential) in data-sensitive environments, taking pride in catching edge cases before code touches production. LLM Evaluation & Red Teaming: You have hands-on experience — or strong hands-on interest — evaluating LLM/RAG systems with tools like RAGAS or DeepEval, and adversarially testing them with frameworks like DeepTeam, PyRIT, or similar red-teaming tools. CI/CD Integration Skills: You understand modern DevOps workflows and feel comfortable embedding automated regression, evaluation, and red-team suites directly into GitHub Actions or GitLab CI pipelines. AI Tools: Hands-on experience leveraging AI tools (e.g., Claude, Cursor) in daily engineering activities, including test design, automation development, debugging, root cause analysis, documentation, and test maintenance. MLOps/LLMOPs Familiarity: Basic understanding of the ML/LLM lifecycle — prompt and model versioning, evaluation dataset management, and monitoring AI systems in production. Craftsmanship & Clear Communication: You value complete task ownership, care about the business logic behind the product, and communicate transparently in English with zero tolerance for rudeness. Can get you a few bonus points: Fintech & Payment Systems Literacy: Prior experience working with financial APIs, payment processing flows, or data security and compliance standards. Protocol Breadth: Hands-on exposure testing GraphQL or gRPC transport layers alongside traditional REST endpoints. LLMOps & Observability: Familiarity with LLM observability/tracing tools (e.g., LangSmith, Langfuse, Arize) or experiment tracking tools. Performance & Cloud Exposure: Familiarity with performance or load testing concepts and basic working experience with cloud infrastructure (AWS, GCP, or Azure). The UMO Standard: Dynamic Work Environment: Vibrant offices and strong local teams across Lisbon, Dubai, Kyiv, and Lviv — with openness to remote for the right fit. Rest & Recovery: 24 days of annual leave, dedicated paid sick leave, and observance of Public Holidays to keep you at your best. Recharge Week: After your first year with us, enjoy two consecutive 4-day work weeks annually — a built-in reset to help you sustain peak performance for the long run. Full Setup: Top-of-the-line hardware and a home office stipend so you can do your best work anywhere. Growth Ecosystem: A dedicated learning budget and a clear, accelerated path toward executive-level leadership within the UMO ecosystem. Impact & Ownership: Your work directly shapes the product and company direction — not just executes it.