Coles Group logo

AI Engineer (Evaluations) - Hawthorn East

Hiring from
Australia
Work type
Hybrid
Posted

626,802 remote jobs, straight from company career pages

100% free · New jobs every minute

Show job description

We’ve been proudly serving Australian communities since 1914 and have grown to become a top 30-listed company on the ASX with a portfolio of iconic homegrown brands. At the heart of our continued success is over 115,000 team members working across the country, all contributing to bringing our customers smiles. You’ll not only get to see your work make a difference, you’ll be part of a team working together to help Australians eat and live better every day.

About the team

Our Technology team is the backbone of the business, building and maintaining the information technology infrastructure that support Coles Group. As a part of the Technology team, you get the opportunity to work through complex, challenging, meaningful problems using large data sets and new technologies to expand your skills and experience.

About the role

As the AI Engineer (Evaluations) in Coles’ AI Commerce team, you’ll be responsible for how we measure the quality of the AI agent in the Coles website and apps that helps customers find products, build their basket, manage orders and sort out issues.

This is an AI engineering role focused on evaluation. You’ll build and manage our evals, keep them trustworthy, and turn the results into clear actions for the team. When the agent’s quality shifts, you’ll be the first to know why. This is a hands-on role. You’ll work AI-first, using agentic development tools to ship high-quality work at pace. You’ll work closely with the Technical Lead, AI engineers, product managers and designers to make sure what we build balances UX, quality and safety.

If you want to shape the quality of a production AI agent used by customers across the nation, with the scale and backing of a major retailer behind you, this is that role.

Key responsibilities will include:

  • Build and maintain the evaluation framework for the AI Commerce Agent: golden datasets, test scenarios and LLM-as-judge metrics
  • Run evals against every significant change and make the results a clear go/no-go signal for release
  • Analyse results to find failure patterns, root causes and regressions, and turn them into clear, prioritised actions for engineers and product
  • Calibrate LLM judges against human review so the scores can be trusted
  • Grow coverage as the agent grows, including edge cases, safety and adversarial (red-team) scenarios
  • Help build online evaluation from production conversations, tracing and customer feedback
  • Tie eval metrics to product success metrics and report on agent quality to the team and stakeholders
  • Use agentic development tools to ship high-quality code at pace
  • Stay across developments in AI evaluation and bring what’s useful into our approach

About you and your skills:

We're looking for a sharp, curious engineer who cares about getting quality right. You're rigorous with data, you don't trust a number until you understand it, and you can explain what the results mean and what to do next. You take ownership of your work, learn quickly and back your views with evidence. You work well in a team and you're comfortable pushing back when the data says something isn't ready.

You'll bring:

  • Proven (2 to 3 years) experience in AI engineering, ML or data science, including hands-on work evaluating LLM or agentic systems
  • Strong Python and SQL
  • Hands-on experience with LLM evaluation: golden datasets, LLM-as-judge, rubric design and tracing
  • A solid grounding in statistics and experiment design, such as sampling, significance and inter-rater agreement
  • An understanding of how AI agents work: tool use, orchestration, RAG and guardrails
  • AI-first ways of working. Agentic coding tools are core to how you build, and you know how to use them without compromising quality
  • Clear communication. You can turn complex results into a story that engineers and stakeholders act on
  • Pragmatism. You apply the 80/20 rule and know when good enough is right for now and when quality is non-negotiable

What will make you stand out:

  • Experience with red-teaming or AI safety testing
  • Experience with evaluation tooling or observability platforms, such as Azure AI Foundry, App Insights or similar
  • A visible footprint in the AI engineering community, such as open source, papers, talks or technical writing
  • Experience in e-commerce, search, recommendations or conversational commerce
  • Show us what you've built. Links to GitHub, shipped products or your writing are welcome with your application.

What’s in it for you?

  • Flexible working options: We know that work is only one part of your life, so we actively encourage a positive work-life balance and provide hybrid working options to help you achieve it.
  • Office perks: Take advantage of our gym facility and fitness classes, free parking, BBQ area, mini-Coles supermarket, fooderie hub where you can sample new products before they hit the shelves, school holiday program and so much more when you come in.
  • Discounts: Eligible team members receive 5% discount all year round on your Supermarket and Liquor online and in-store purchases. We also offer additional periods of double discount (10%) at various times throughout the year. As well as hotdeals exclusive to team members that translate into additional savings.
  • Reward through recognition: Give and receive recognition, linked to our Coles values, through our digital recognition platform ‘mythanks’. You can accumulate points to redeem in the online shop for exciting gifts and electronic gift cards from an extensive range of retailers.
  • Opportunities for learning and development: No matter where you start within our diverse business, you’ll have experiences, exposure, and education to satisfy you. Discover and explore a variety of career development programs and job-specific training.
  • Paid parental leave: We understand how important your life outside work is and offer permanent team members paid parental leave to support you in balancing work and family.
  • Investment in your future: Our annual team member share plan offer allows eligible team members to make regular pre-tax salary sacrifice deductions to purchase Coles Group shares.

About the recruitment process

We’re continuing to build a gender equitable team, and a culture that is just as diverse, inclusive and welcoming as the communities we serve. We are committed to creating a workplace that is safe and respectful for our team. We encourage applications from people of all ages, cultures (including Aboriginal and Torres Strait Islander peoples), abilities, sexual orientation and gender identities.

We’re happy to adjust our recruitment process to support candidates with disability. For further information and additional contact details visit the ‘Our Recruitment Process’ section of our careers site or email inclusionrecruitment@coles.com.au

Job ID: 200884

Employment Type: Full time

LI-MM1

Apply for this job100% free