AI QA / 100% Remote @BRAZIL / USD payment / Advanced English (must)

AI QA / 100% Remote @BRAZIL / USD payment / Advanced English (must) Apply now Company Description Bloom Talent Partners specializes in IT recruitment, focusing primarily on the Latam market. We bridge the gap between employers and exceptional IT professionals, ensuring a perfect cultural fit. For companies, we provide tailored IT talent, tech expertise, efficient hiring, and a personalized approach. For candidates, we offer tailored growth, tech insights, an effortless search, and supportive guidance to align with their career goals. Role Description Position: AI QA Location: Brazil About the role: As the AI QA, you will be the analytical architect of the quality and performance ecosystem that powers Made. You will be responsible for answering the critical question, “How do we measure the quality of a generative AI’s output at scale?” This role is not about traditional QA, it is about creating a new discipline. You will design the frameworks, metrics, and systems that ensure our AI agents are not only responsive but also strategically valuable, creatively inspiring, and personality-aligned. You will be a key leader in a new function, working at the intersection of data science, product strategy, and creative insight to build a truly revolutionary product. Key Responsibilities: ● Design and implement a comprehensive AI quality framework from the ground up, blending automated testing with expert human evaluation. ● Develop detailed Quality Scorecards to systematically assess the performance of our AI agents and their skills, such as content ideation or comment replies. ● Codify and maintain the distinct personalities and communication styles of our AI agents. ● Analyze and interpret the outputs from our Automated Quality Engine, identifying systemic trends, model weaknesses, and opportunities for improvement. ● Collaborate closely with our Creator-in-Residence (CIR) to translate their qualitative, expert feedback into actionable data and new test cases for the automated system. ● Partner with Product and Engineering teams to provide clear, data-driven feedback that directly informs model fine-tuning and the development of new AI skills. ● Own the curation and management of our "Golden Datasets", the benchmark collections of ideal prompts and responses that define excellence for our platform. Key Requirements: ● Bachelor’s degree in Computer Science, Information Systems, Computer Engineering, or a related field. ● Minimum of 3 years of experience with QA ● Experience with web application testing frameworks and tools (e.g., Jest, Cypress, Selenium). ● Experience with automated tests in Python. ● Experience with LangChain, LangGraph or similar tools. LangSmith is a plus. ● Experience with RAG and fine-tuning of LLMs. ● Experience in Git and code reviews. ● Understanding of database systems (e.g., MySQL, PostgreSQL, MongoDB) and writing optimized SQL queries. ● Understanding of A/B testing, data analysis, and performance metrics. ● Exceptional problem-solving abilities and attention to detail. ● Effective communication skills and the ability to work in a fast-paced, collaborative environment. Plus requirements: ● Familiarity with evaluation metrics for generative AI (e.g., coherence, factuality, diversity, style alignment, and hallucination detection). ● Experience with machine learning libraries such as TensorFlow or PyTorch. ● Experience in Python for web development. ● Experience with FastAPI. Locations BRAZIL Remote status Fully Remote