Senior AI Tools Testing Engineer
- Salary
- $120K–$140K
- Hiring from
- United States
- Work type
- Hybrid
- Posted
621,192 remote jobs, straight from company career pages
100% free · New jobs every minute
Show job descriptionHide job description
Job Title: Senior / Lead AI Tools & Testing Engineer (Agentic AI & LLM Evaluation)
Location: Dallas, TX (Hybrid)
Duration: Long‑Term
Experience:10 + Years
We seek a senior/lead AI engineer with hands‑on experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. This role focuses on creating reliable, scalable, and production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.
The candidate will collaborate with engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications.
Key Responsibilities
1. Agentic AI & GenAI Engineering
- Design, develop, and deploy production‑grade agentic and generative AI applications.
- Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
- Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
- Create scalable LLM‑powered applications and reusable AI components.
2. RAG & LLM Application Development
- Design and implement Retrieval‑Augmented Generation (RAG) solutions.
- Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
- Optimize retrieval accuracy, response quality, latency, and cost.
- Apply grounding, hallucination reduction, and context‑relevance techniques.
3. LLM Evaluation & AI Testing
- Build evaluation frameworks for LLM‑based applications and agentic workflows.
- Automate evaluation with DeepEval and other LLM evaluation tools.
- Use LLM‑as‑a‑Judge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
- Create evaluation datasets, test scenarios, benchmarks, and regression‑testing pipelines.
- Evaluate agent behavior, tool execution, workflow accuracy, and response consistency.
- Identify failure patterns and continuously improve AI application quality.
4. AI Observability & Monitoring
- Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
- Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
- Establish debugging mechanisms for complex multi‑step agentic workflows.
- Develop monitoring practices to detect performance degradation and operational issues.
5. AWS AI Platform & Production Engineering
- Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
- Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
- Apply production‑readiness practices covering scalability, reliability, security, and performance.
- Support CI/CD integration and automated evaluation of AI applications.
- Optimize model selection, inference costs, latency, and resource utilization.
6. Technical Leadership & Collaboration
- Provide technical guidance to engineering and QA teams.
- Establish reusable frameworks, implementation patterns, and best practices.
- Collaborate with architecture, DevOps, QA, and engineering teams.
- Participate in design discussions, code reviews, proof‑of‑concept development, and implementation.
- Mentor engineers and help teams adopt effective AI development and testing practices.
Required Technical Skills
Agentic AI & Orchestration
- LangGraph
- LangChain
- Amazon Bedrock
- Amazon Bedrock AgentCore
- AI agents, multi‑agent orchestration, tool calling, workflow management
Generative AI & RAG
- Large Language Models (LLMs)
- Retrieval‑Augmented Generation (RAG)
- Embeddings and vector search
- Prompt engineering and context management
- LLM application development
LLM Evaluation & Testing
- DeepEval
- LLM‑as‑a‑Judge techniques
- Evaluation metrics and benchmarking
- Automated evaluation and regression testing
- Assessment of hallucination, faithfulness, relevance, and response quality
AI Observability & Monitoring
- OpenTelemetry
- LangSmith
- Langfuse
- LLM tracing, debugging, and performance monitoring
Production Engineering
- Deploying and operating production‑grade GenAI applications
- API integration and backend engineering
- CI/CD and automated testing integration
- Scalability, reliability, security, and performance optimization
Preferred Qualifications
- 8+ years of software engineering or IT experience.
- Hands‑on experience developing and deploying GenAI or Agentic AI applications.
- Experience with enterprise‑scale AI implementations.
- Familiarity with AWS cloud services and enterprise AI security practices.
- Experience establishing AI quality standards and evaluation methodologies.
- Strong problem‑solving, communication, and technical collaboration skills.
What Success Looks Like
- Development and deployment of production‑grade agentic AI applications.
- Reliable orchestration and execution of AI agents and workflows.
- Effective LLM evaluation and automated AI testing frameworks.
- Improved AI application accuracy, reliability, observability, and performance.
- Reusable engineering practices for building and maintaining enterprise GenAI solutions.
Important: This is a hands‑on engineering position. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.
Location: Dallas, TX (Hybrid)
Duration: Long‑Term
Experience:10 + Years
We seek a senior/lead AI engineer with hands‑on experience building, orchestrating, monitoring, and evaluating production‑grade Generative AI and Agentic AI applications. The ideal candidate is expert in Amazon Bedrock, AgentCore, LangGraph, LangChain, RAG, LLM observability, and evaluation frameworks. This role focuses on creating reliable, scalable, and production‑ready AI applications, emphasizing agent orchestration, AI testing, LLM evaluation, and continuous quality monitoring.
The candidate will collaborate with engineering, QA, and platform teams to establish best practices for building, testing, monitoring, and improving AI‑powered applications.
Key Responsibilities
1. Agentic AI & GenAI Engineering
- Design, develop, and deploy production‑grade agentic and generative AI applications.
- Build and orchestrate AI agents using LangGraph, LangChain, Amazon Bedrock, and AgentCore.
- Implement multi‑agent workflows, tool calling, state management, memory, and orchestration.
- Create scalable LLM‑powered applications and reusable AI components.
2. RAG & LLM Application Development
- Design and implement Retrieval‑Augmented Generation (RAG) solutions.
- Develop pipelines for document ingestion, chunking, embedding, retrieval, reranking, and context management.
- Optimize retrieval accuracy, response quality, latency, and cost.
- Apply grounding, hallucination reduction, and context‑relevance techniques.
3. LLM Evaluation & AI Testing
- Build evaluation frameworks for LLM‑based applications and agentic workflows.
- Automate evaluation with DeepEval and other LLM evaluation tools.
- Use LLM‑as‑a‑Judge methods to assess response quality, relevance, accuracy, faithfulness, and reasoning.
- Create evaluation datasets, test scenarios, benchmarks, and regression‑testing pipelines.
- Evaluate agent behavior, tool execution, workflow accuracy, and response consistency.
- Identify failure patterns and continuously improve AI application quality.
4. AI Observability & Monitoring
- Implement tracing, monitoring, and observability with OpenTelemetry, LangSmith, and Langfuse.
- Track LLM requests, agent execution flows, tool calls, latency, token consumption, errors, and performance.
- Establish debugging mechanisms for complex multi‑step agentic workflows.
- Develop monitoring practices to detect performance degradation and operational issues.
5. AWS AI Platform & Production Engineering
- Deploy AI applications using Amazon Bedrock and Bedrock AgentCore.
- Integrate foundation models, agent frameworks, APIs, and enterprise data sources.
- Apply production‑readiness practices covering scalability, reliability, security, and performance.
- Support CI/CD integration and automated evaluation of AI applications.
- Optimize model selection, inference costs, latency, and resource utilization.
6. Technical Leadership & Collaboration
- Provide technical guidance to engineering and QA teams.
- Establish reusable frameworks, implementation patterns, and best practices.
- Collaborate with architecture, DevOps, QA, and engineering teams.
- Participate in design discussions, code reviews, proof‑of‑concept development, and implementation.
- Mentor engineers and help teams adopt effective AI development and testing practices.
Required Technical Skills
Agentic AI & Orchestration
- LangGraph
- LangChain
- Amazon Bedrock
- Amazon Bedrock AgentCore
- AI agents, multi‑agent orchestration, tool calling, workflow management
Generative AI & RAG
- Large Language Models (LLMs)
- Retrieval‑Augmented Generation (RAG)
- Embeddings and vector search
- Prompt engineering and context management
- LLM application development
LLM Evaluation & Testing
- DeepEval
- LLM‑as‑a‑Judge techniques
- Evaluation metrics and benchmarking
- Automated evaluation and regression testing
- Assessment of hallucination, faithfulness, relevance, and response quality
AI Observability & Monitoring
- OpenTelemetry
- LangSmith
- Langfuse
- LLM tracing, debugging, and performance monitoring
Production Engineering
- Deploying and operating production‑grade GenAI applications
- API integration and backend engineering
- CI/CD and automated testing integration
- Scalability, reliability, security, and performance optimization
Preferred Qualifications
- 8+ years of software engineering or IT experience.
- Hands‑on experience developing and deploying GenAI or Agentic AI applications.
- Experience with enterprise‑scale AI implementations.
- Familiarity with AWS cloud services and enterprise AI security practices.
- Experience establishing AI quality standards and evaluation methodologies.
- Strong problem‑solving, communication, and technical collaboration skills.
What Success Looks Like
- Development and deployment of production‑grade agentic AI applications.
- Reliable orchestration and execution of AI agents and workflows.
- Effective LLM evaluation and automated AI testing frameworks.
- Improved AI application accuracy, reliability, observability, and performance.
- Reusable engineering practices for building and maintaining enterprise GenAI solutions.
Important: This is a hands‑on engineering position. Candidates with practical experience building and evaluating AI applications are preferred over those whose experience is limited to AI strategy, architecture documentation, conventional QA automation, or using AI coding assistants.
Flexible work from home options available.
Compensation: $120,000.00 - $140,000.00 per year
About Us
We work to deliver profitability in your business – with effective communication, consulting, and interactive solutions. Following an Agile Work Approach, we make sure you get the ideal solutions at minimum expenses.
Work Approach
Our Philosophy
Our Philosophy starts-and-ends at the Client-first approach. Be it understanding your business requirements to choosing the right technologies, we work as a collective team that takes all the possible steps to grow continuously towards our common goal.
Our Philosophy starts-and-ends at the Client-first approach. Be it understanding your business requirements to choosing the right technologies, we work as a collective team that takes all the possible steps to grow continuously towards our common goal.
Work Policy
We promote a collaborative work environment. We involve everyone working in the organization in community decisions and encourage them to think from a broader perspective. Our work process promotes flexibility and we maintain a high level of discipline at different levels of execution.
The Future
SelectMinds have years of experience in the domain helps us understand the need-of-the-hour better. This understanding drives us to a better future with every minute ticking. We believe we will be taking off major businesses from their flagship positions, with the products we are eyeing today.
