Principal Data Engineer
- Hiring from
- United States
- Work type
- Remote
- Posted
625,790 remote jobs, straight from company career pages
100% free · New jobs every minute
Show job descriptionHide job description
Responsibilities
We are looking for a Principal Data Engineer who brings together software engineering and data engineering to help build the next generation of our data platform.
This is not a traditional data engineering role focused primarily on building and maintaining individual ETL pipelines. You will design and build reusable data integration capabilities, frameworks, connectors, and adapters that make it easier to bring data from many different enterprise systems into our platform.
You will work within our platform engineering organization to build scalable capabilities that connect databases, APIs, event streams, and other enterprise data sources into Microsoft Fabric, where that data can support downstream applications, AI capabilities, and knowledge graph experiences.
The right person for this role thinks like a software engineer building a data platform: creating reusable patterns, designing for different integration scenarios, writing production-quality code, and considering scalability, reliability, monitoring, and maintainability from the start.
What You'll Do
- Design and build reusable data integration frameworks, connectors, and adapters that bring data from multiple enterprise systems into Microsoft Fabric.
- Build scalable ingestion capabilities for databases, REST APIs, event streams, files, and other integration patterns.
- Develop production-quality software in Python and PySpark, with an emphasis on reusable, modular, maintainable code.
- Design integration capabilities that can support many data sources rather than solving for a single pipeline or use case.
- Build and operate robust ETL/ELT and CDC capabilities for structured and semi-structured enterprise data.
- Implement database mirroring and replication patterns for systems such as SQL Server and Oracle.
- Create resilient integration patterns that address connection failures, retries, schema changes, data transformations, and replay.
- Build observability into the platform through monitoring, alerting, logging, lineage, and data-quality controls.
- Apply software engineering practices including version control, automated testing, CI/CD, code reviews, and design patterns.
- Contribute to Platform Studio's data engineering capabilities, helping determine which connectors, integration patterns, and frameworks should become reusable product capabilities.
- Work with Graph Engineering and AI teams to make enterprise data available for downstream knowledge graph, application, and AI use cases.
Education / Qualifications
- 7+ years in data engineering, with hands-on production experience building and operating ETL/ELT pipelines at scale
- Strong proficiency with Microsoft Fabric or Azure Synapse Analytics — Delta Lake, lakehouse architecture, and Spark-based transformation
- Experience with database mirroring or CDC for SQL Server (e.g. Fabric Database Mirroring, Debezium, SQL Server replication) and Oracle
- Hands-on experience with REST API integration and OData/OpenAPI-based data extraction
- Solid understanding of medallion/lakehouse architecture patterns (Bronze/Silver/Gold or equivalent staging discipline)
- Experience with pipeline orchestration tools (Azure Data Factory, Fabric Data Workflows, Airflow, or comparable)
- Proficiency in Python and/or PySpark for transformation logic and notebook-based ETL
- Strong data quality mindset — schema validation, null handling, deduplication, and monitoring at ingestion
- Familiarity with enterprise authentication patterns (OAuth2 client credentials, service principals, managed identities)
Preferred:
- Experience with Databridge Pro or comparable industrial middleware integration products
- Background in capital projects, EPC, or industrial enterprise software data models
- Exposure to knowledge graph construction pipelines downstream of lakehouse staging
About Octave
Octave provides mission-critical software that empowers organizations to make informed decisions across every stage of the asset lifecycle - Design, Build, Operate and Protect - where performance, safety, and reliability are non-negotiable and failure is not an option.
Turning complex operational data into actionable intelligence, Octave connects expertise, real-world conditions and enterprise-scale insight to improve performance, resilience and incident response where it matters most.
Octave has more than 7,000 employees in 45 countries. Learn more at octave.com and follow us on LinkedIn.
Why work for Octave?
All in. Always forward. That's the way we do things around here. We put trust in our people because we believe it's the best way to unleash potential, bring ideas to life, and keep moving ahead. And it's why we're committed to creating an environment that's truly supportive, providing you with the resources you need to support your ambitions, no matter who you are or where you are in the world.
Everyone is welcome
At Octave, we believe that diverse and inclusive teams are critical to the success of our people and our business. Here, everyone is welcome. As an inclusive workplace, we don't discriminate. In fact, we embrace differences and are fully committed to creating equal opportunities, an inclusive environment, and fairness for all.
Respect is the cornerstone of how we operate, so speak up and be yourself. You're valued here.
Recruitment Fraud Alert
Octave posts all official job opportunities on either https://careers.octave.com/ or https://www.octave.com/about/careers and communicates only from email addresses ending in @octave.com. We never request payment or personal banking information during recruitment. No offers will ever be extended without a proper interview via Teams or in person, never done over email alone. If you suspect fraud, it probably is, and contact us at careers@octave.com
