Svitla Systems Inc. is looking for a Senior AI Engineer for a full-time position (40 hours per week) in Argentina. Our client is a developer of an AI-native project intelligence platform for residential construction - the source of truth that helps architects, builders, and GCs manage design plans, decisions, approvals, and change orders across the full project life cycle. The platform includes an AI assistant that automates manual admin tasks, answers project and document questions, and logs decisions and action items in a transparent audit trail.
The AI-native stack is built from scratch on Azure, spanning a mobile app and a desktop experience, and uses retrieval (RAG) and an agentic orchestration engine. The in-house team includes a CTO with a data science/AI background, a head of product design (ex-Amazon global UX), a small senior dev team, and an orchestration engine built by a technical advisor.
You'll assume prompts and models will drift, that a new model release can quietly break what worked yesterday, and that you need objective evidence - not gut feel - to know whether the AI assistant is getting better or worse. The instinct is to build the test harness, the golden datasets, and the drift reports that let the team ship changes with confidence. Success looks like: fewer surprises in production, safe model upgrades, and a measurable, improving quality bar.
Requirements
Senior-level experience in prompt engineering on production LLM products.
Hands-on experience building LLM evaluation / eval frameworks (offline and/or online), including regression and A/B testing.
Strong intuition for measuring model quality - designing metrics and test sets, not just eyeballing outputs.
Strong knowledge of Python.
Be comfortable integrating with an Azure-based AI stack and CI-style automation for eval runs.
Responsibilities
Design, test, and maintain production prompts across a range of nuanced workflows and use cases on both mobile and desktop.
Turn fuzzy product requirements into reliable, structured prompt behavior - system prompts, tool-use instructions, output schemas, and guardrails.
Understand modern LLM behavior in depth: context management, structured outputs, function/tool calling, latency vs. accuracy trade-offs, and cost/token efficiency.
Work hands-on within an Azure-based AI stack and collaborate closely with the CTO, product design, and existing engineers.
Onboard fast and ship - comfortable in a startup with a tight, senior team moving at speed
Own prompt engineering and stand up the evaluation framework that keeps the product reliable as models, prompts, and features change. You should make sure it catches quality regressions and model drift before customers do.
Design an evaluation harness for prompts, agents, and retrieval - golden datasets, task-level metrics, and automated scoring for accuracy, faithfulness, and regression.
Guard against model drift: implement regression and A/B testing so that adopting a new model version (e.g., a new GPT/Claude release) is a measured, safe decision rather than a surprise.
Stand up an offline evaluation pipeline that mirrors production workflows and produces a scheduled report (e.g., nightly) flagging model variation, drift, or quality drops.
Feed eval results back into the prompt and workflow improvements, and give the team objective evidence of what to change.
Nice To Have
Background in data science/statistics (helpful for meaningful metrics and drift detection).
Familiarity with RAG and agentic workflows - enough to evaluate them well.
Experience with observability/LLM monitoring tooling.
WE OFFER
US and EU projects based on advanced technologies.
Competitive compensation based on skills and experience.
Comprehensive private medical insurance.
Regular performance appraisals to support your growth.
Flexibility in workspace, either remote, our welcoming office or local coworking.
Bonuses for recommendations of new employees.
Bonuses for article writing, public talks, other activities.
15 vacation days, 10 national holidays, 10 sick leaves.
Personalized learning program tailored to your interests and skill development.
Free tech webinars and meetups organized by Svitla.
Fun corporate online\offline celebrations and activities.