Intuitive.ai logo

Senior Data Engineer

Intuitive.aiApplies on LinkedInData & Analytics
Hiring from
Canada
Work type
Remote
Posted

624,711 remote jobs, straight from company career pages

100% free · New jobs every minute

Show job description

About us:

Intuitive.AI is one of the fastest-growing (INC 5000, CRN) Cloud & SDx solution and services companies supporting enterprise customers on a global scale. Intuitive is an "Engineering Company" delivering measurable value and key business outcomes.

Intuitive Superpowers:

- DataOps & AI/ML

- Cloud Native, AppSecOps, DevSecOps

- Cloud Migration & Transformation

- Cloud FinOps

- Cybersecurity (App/Data/Infra) & GRC

- SDx & Digital Workspace


We are proud to partner with some of the world's leading enterprises and serve 200+ customers across different industry verticals. We have achieved many milestones along the way, including being recognized as a top-10 fast-growth 150 IT company in the Americas by CRN in 2022 and being named one of America's fastest-growing private companies by INC 5000 in 2022. That’s not all! Even CIO Review awarded us as the Most Promising Cloud Migration Company and Artificial Intelligence Solutions Provider in 2022.


About the job:

Title: Senior Data Engineer

Location: Remote across USA/ Canada

Work Model: Remote

Employment Type: Full-Time/ Contract


About the Role:


We're building a data platform that turns raw operational data from databases, APIs, SaaS systems, and event streams into trusted, well-modelled datasets for analytics, reporting, and AI applications. We need a hands-on Senior Data Engineer who is strong in Python and SQL and has built real production pipelines: ingestion, modelling, transformation, orchestration, and quality, on a modern cloud stack built around Snowflake — SQL, Snowpark Python, and Snowflake Notebooks — with Spark where large-scale processing calls for it.

This is a hands-on technical role. You will spend most of your time writing Python and SQL, exploring and profiling data in notebooks, shipping pipelines, and operating what you ship. You will own datasets end-to-end, from source analysis through to the tables and views that analysts, dashboards, and AI applications depend on.


Key Responsibilities

As a Senior Data Engineer, you will:

  • Build and operate production data pipelines in Python and SQL that ingest data from databases, APIs, files, SaaS sources, and event streams, in both batch and incremental or streaming modes.
  • Own datasets end-to-end — from source analysis and ingestion through modelling, transformation, testing, rollout, and production operation. You own the dataset from raw landing zone to consumer-facing table, and you answer the page when it breaks.
  • Explore, profile, and analyse data in notebooks — use Snowflake Notebooks or Jupyter with Snowpark, pandas, or Polars to understand new sources, validate assumptions, prototype transformations, and investigate data issues before and after they reach production.
  • Design and maintain data models — layered raw / staging / curated structures, dimensional and wide-table models, clear keys and grain, and schema evolution that doesn't break consumers.
  • Develop transformations in SQL and dbt (or an equivalent framework) with tests, documentation, and lineage, so every curated table has a clear owner, contract, and definition.
  • Orchestrate pipelines with Airflow or a similar scheduler — dependencies, retries, backfills, SLAs, and alerting — so runs are predictable and recoverable.
  • Work deeply in Snowflake — warehouse schemas, roles and access, Snowpark Python, loading and incremental patterns, performance tuning, and cost awareness, with Spark for large-scale processing where Snowflake alone isn't the right tool.
  • Implement data-quality controls — validation, freshness checks, reconciliation against sources, anomaly detection, and failure handling that alerts before consumers notice.
  • Operate pipelines in production — logs, metrics, dashboards, actionable alerts, runbooks, and disciplined incident investigation and post-incident fixes.
  • Define and maintain schemas and data contracts so analysts, dashboards, APIs, and AI applications can rely on stable, well-understood interfaces while sources evolve.
  • Ship through CI/CD — automated tests for pipelines, transformations, and contracts; environment promotion; and release validation for data changes.
  • Prepare analysis-ready and AI-ready datasets — curated tables, aggregates, and feature-style views that analysts, BI tools, and AI applications can consume safely.
  • Raise the engineering bar — write clean, well-tested Python and SQL; review others' work; document data flows and decisions with concise diagrams and notes; and mentor engineers on modelling, testing, and operability.
  • Use AI-assisted engineering tools effectively to accelerate development, testing, review, documentation, and debugging, while validating generated output.
  • Apply shift-left security and governance — data classification, PII handling, and least-privilege access considered during implementation, not as a final release check.


Required Qualifications

  • 6+ years of data engineering or software engineering experience, with 2+ years in senior roles building and operating production data pipelines.
  • 2+ years running data pipelines in production — you've owned pipelines past the initial ship: on-call, incident response, backfills, schema migrations, and the operational grind of keeping datasets fresh and correct for real consumers.
  • Strong Python — you write clear, idiomatic, well-tested Python for data pipelines, transformations, and tooling, with hands-on depth in pandas or Polars and the standard data stack (typing, packaging, pytest, virtual environments).
  • Strong SQL — complex analytical queries, window functions, CTEs, query optimisation, and the ability to read a query plan and fix a slow query.
  • Notebook-driven data analysis — comfortable in Snowflake Notebooks or Jupyter for exploratory data analysis, profiling, data validation, and prototyping, and disciplined about turning notebook work into production code.
  • Strong hands-on Snowflake experience — schemas, roles and access control, Snowpark Python, loading and incremental patterns, performance tuning, and cost awareness.
  • Experience with dbt or an equivalent SQL transformation framework — models, tests, documentation, and environment promotion.
  • Experience with a workflow orchestrator — Airflow, Dagster, Prefect, or similar — including scheduling, dependencies, retries, backfills, and alerting.
  • Strong data modelling skills — dimensional and wide-table modelling, layered warehouse / medallion architectures, schema evolution, and data contracts.
  • Production experience with both batch and incremental or streaming pipelines, with a working understanding of idempotency, deduplication, replay, late-arriving data, and recovery.
  • Hands-on experience with at least one major cloud platform (AWS, GCP, or Azure) and its storage, identity, and data services.
  • Experience building observable pipelines — logs, metrics, freshness checks, data-quality tests, dashboards, and actionable alerts.
  • Experience delivering changes through CI/CD with automated tests for pipelines, transformations, and contracts.
  • Experience using AI-assisted engineering tools as part of routine delivery, with the judgment to validate and improve generated output.


Preferred Qualifications

  • Apache Spark or PySpark experience for large-scale batch and streaming processing.
  • Databricks exposure — Delta Lake and Unity Catalog.
  • Apache Kafka, Amazon Kinesis, or equivalent event-streaming platforms in production.
  • Open table formats — Apache Iceberg or Delta Lake.
  • Statistical analysis or data-science fundamentals — distributions, outlier detection, basic hypothesis testing — applied to data validation and investigation.
  • Experience preparing datasets for machine learning, AI agents, Retrieval-Augmented Generation (RAG), or analytical APIs.
  • Experience with data-quality and observability tooling such as Great Expectations, Soda, or Monte Carlo.
  • Infrastructure-as-code experience (Terraform, OpenTofu, or similar) for data-platform resources.
  • Experience handling PII or sensitive data — masking, tokenisation, row-level and column-level controls.
  • Experience optimising Snowflake, Spark, and object-storage workloads for performance and cost.
  • Experience mentoring engineers or leading technical design discussions.


First 30 / 60 / 90 Days

  • First 30 days — Build context on the sources, consumers, and existing pipelines; profile the key datasets in notebooks; meet the analytics, product, and AI teams you serve; ship your first production change.
  • First 60 days — Own one dataset end-to-end: a clear source-to-consumer plan, defined freshness and quality targets, and one pipeline carried from design through to production with monitoring and a runbook.
  • First 90 days — Take a cross-team dataset from design to adoption and establish durable patterns — modelling conventions, contracts, testing, and operational standards — that other engineers follow.


Apply for this job100% free