Senior AI Data Engineer | LATAM

About Cadre AI

The Role

What You’ll Do

Lakehouse and Pipeline Engineering

• Design and build medallion (bronze, silver, gold) lakehouses on Databricks (Delta Lake, Unity Catalog, Lakeflow Spark Declarative Pipelines) and Microsoft Fabric (OneLake, Data Factory), and on Snowflake where a client already runs it.
• Own ingestion from ERPs, CRMs, and SaaS platforms using platform-native connectors (Lakeflow Connect, Fabric Data Factory), Fivetran, or dlt, including change data capture and schema drift handling.
• Model and transform data with dbt, orchestrate it with Apache Airflow, Dagster, or Lakeflow Jobs, and ship it through CI/CD with GitHub Actions and Declarative Automation Bundles (formerly Databricks Asset Bundles).
• Build data quality checks and data contracts into every pipeline so downstream agents never consume silent errors.
• Run agentic data engineering safely in client Databricks repositories: set up coding agents (Claude Code, Codex) with shared skills, policies, and guardrails, using Omnigent where it is available, and accelerate legacy-to-Databricks migrations with Genie Migrate.

Retrieval Systems: RAG and GraphRAG

• Build production RAG pipelines end to end: document parsing and OCR, chunking, embeddings, hybrid retrieval (BM25 plus dense vectors), and cross-encoder reranking.
• Design GraphRAG systems that extract entities and relationships into a knowledge graph, using frameworks such as Microsoft GraphRAG or LightRAG on graph stores such as Neo4j, for questions that span documents, systems, and multiple entity hops.
• Choose the right retrieval architecture for each use case, from hybrid search with reranking to graph-based and agentic retrieval, and justify the cost and maintenance burden of each to the client.
• Operate vector, full-text, and hybrid search infrastructure (Databricks AI Search, formerly Vector Search; Azure AI Search; OpenSearch; pgvector), including incremental indexing, permission-aware retrieval, and citations back to the source.
• Build and deploy retrieval-backed agents on Databricks Agent Bricks, using its code-first tooling (Agent Bricks CLI, Agent Runtime on Databricks Apps) and governing model access through Unity AI Gateway.

Semantic Layers and Business Ontologies

• Translate how a client defines a customer, an order, or an invoice into governed metric and entity definitions using dbt Semantic Layer (MetricFlow), Power BI semantic models, Cube, Databricks metric views, or Snowflake semantic views.
• Lead data readiness assessments: inventory source systems, map fields and business logic, and document gaps and remediation steps before engineering starts.
• Expose governed data to Claude and other agents through MCP, so every agent, dashboard, and text-to-SQL query answers from the same definitions.
• Use AI to draft documentation and definitions, then own their correctness together with the business owner.

Evaluation Data and Model Quality

• Build golden datasets, output and label pairs, and backtesting splits that measure whether an agent, retriever, or model is right.
• Instrument retrieval and agent pipelines with dedicated tracing and evaluation tools (Langfuse, Arize Phoenix, Braintrust), or with MLflow 3 GenAI evaluation and LLM judges on Databricks, and track recall, precision, and groundedness.
• Track experiments and version models and evaluation runs in the MLflow registry, so every change to a retriever or agent is traceable.
• Design leakage-free, time-aware splits for forecasting, scoring, and risk ranking.
• Monitor drift, freshness, and data gaps that degrade agents in production, and fix them at the data layer.

Governance and Client Delivery

• Implement role-based access, row-level security, and masking in Unity Catalog, Microsoft Purview, or Snowflake roles, and carry those permissions into retrieval and agent context.
• Ensure every data path meets SOC 2 expectations and the client’s own AI governance policies, including the handling of PII, PHI, and payroll data.
• Work directly with client IT, finance, and operations leaders to scope work, secure access, and explain build-versus-buy trade-offs across platforms.
• Record architecture decisions as ADRs in the client repository and contribute reusable pipeline, retrieval, and semantic-layer blocks to Cadre’s internal library.

What You Need to Succeed

• 5+ years of data engineering experience, including at least one production lakehouse you designed and operated on Databricks or Microsoft Fabric with a medallion architecture; experience on both is a plus, since our clients run both.
• A Master’s degree or PhD in Computer Science, Mathematics, Engineering, or a related quantitative field, with the theoretical grounding to read current research and apply it to client problems.
• Hands-on experience shipping RAG pipelines to production (hybrid retrieval on Databricks AI Search or an equivalent engine, reranking, chunking strategy, and permission-aware indexing over messy enterprise documents), with command of the current state of the art: you follow the literature and can explain when hybrid, graph-based, or agentic retrieval fits a problem.
• Experience designing and running GraphRAG or knowledge-graph retrieval (Microsoft GraphRAG, LightRAG, or equivalent, on graph stores such as Neo4j), and the judgment to know when a graph earns its maintenance cost and when hybrid search is enough.
• Production experience orchestrating data pipelines with Apache Airflow or Dagster, including dependencies, backfills, retries, and alerting; you understand the trade-off between task-based and asset-based orchestration and work comfortably in platform-native schedulers such as Lakeflow Jobs and Fabric Data Factory.
• Expert SQL and strong Python and PySpark, with production experience in dbt or platform-native pipelines and in ingestion tools (Fivetran, Lakeflow Connect, Azure or Fabric Data Factory, or dlt); you use coding agents such as Claude Code daily and review everything they generate.
• Experience authoring a semantic or metrics layer (dbt Semantic Layer, Power BI semantic models, Cube, LookML) and turning business definitions into governed metrics that people and agents both rely on.
• Working ML fluency: you understand data leakage, time-based validation, class imbalance, and precision versus recall, and you have built datasets that trained or evaluated a model or retriever, evaluated with Langfuse, Arize Phoenix, Braintrust, or MLflow and versioned in the MLflow registry.
• Hands-on experience building agents on Databricks (Agent Bricks or Mosaic AI Agent Framework) and running coding agents such as Claude Code or Codex under governed harnesses with skills, policies, and guardrails; familiarity with Databricks’ newer agentic tooling (Omnigent, Genie Migrate, Agent Bricks CLI) and with how agents consume data through RAG, text-to-SQL, and MCP.
• Governance depth across Unity Catalog, Purview, or Snowflake RBAC; you treat access control as a design input for pipelines and retrieval alike.
• Client-facing consulting or forward-deployed experience and professional English fluency, because you will work daily with US executives and turn technical trade-offs into business decisions.
• Strong plus: Snowflake Cortex, Palantir Foundry ontologies, RDF/OWL knowledge graphs, DSPy prompt optimization, or LoRA fine-tuning of open-weight models.

Why Cadre AI

• AI-native culture. We do not just build AI for clients. We use it to run our own operations. You will work with people who are as obsessed with the tools as you are.
• Access to the frontier. Through partnerships with Anthropic and OpenAI, you will be among the first to experiment with new models and capabilities.
• Upside. We are bootstrapped, profitable, and growing fast. Growth means new teams and new initiatives, and room to move into them.
• No bureaucracy. Small, scrappy team. Clear accountability. The best idea wins, regardless of who says it.

Compensation