Senior Data Engineer

Job Description This is a remote position. Senior Data Engineer We are seeking a Senior Data Engineer to support a digital regulatory transformation initiative modernizing environmental and natural resource regulatory processes. This role is critical to ensuring that modernization delivers tangible business value by designing, building, and operating reliable data pipelines that ingest and integrate data into modern data management platforms. The ideal candidate will take ownership of data architecture, pipeline development, and data product delivery, collaborating with business stakeholders to understand objectives and translate requirements into scalable, secure solutions that produce trusted, analytics-ready datasets supporting regulatory oversight, compliance monitoring, and evidence-based decision-making. The successful candidate will lead the development of data ingestion, transformation, and integration pipelines using Azure Data Factory, Azure Databricks, and Azure Synapse Analytics. Responsibilities include designing and implementing scalable data architecture on Microsoft Azure, managing data lakes and structured storage using Azure Data Lake Storage Gen2, integrating data from diverse source systems including ServiceNow and geospatial systems, developing robust data models and semantic layers, building and optimizing data workflows using Python and SQL, designing secure data services and APIs using Azure API Management, implementing data governance practices including metadata management and lineage tracking, ensuring compliance with privacy and regulatory standards through role-based access controls and encryption, monitoring and troubleshooting data pipelines for reliability and performance, and leveraging AI and automation tools to streamline workflows and improve development efficiency. Qualifications - 5 years of hands-on experience in Python and SQL for data engineering, applied to developing, orchestrating, and optimizing enterprise-grade ETL/ELT workflows in a large-scale cloud environment - 3 years of experience with Azure services including Storage, SQL, Synapse, and networking for scalable, secure solutions, and with authentication mechanisms such as Service Principals and Managed Identities for secure access in pipelines and integrations - 3 years of hands-on experience with Azure Databricks and Delta Lake, building scalable data pipelines with Databricks Workflows, Jobs, and Notebooks, plus cluster management - 3 years of experience designing data solutions for analytics-ready, trusted datasets using tools like Power BI and Synapse, including semantic layers, data marts, and data products for self-service, data science, and reporting - 4 years of experience using version control systems, specifically Github/Git for version control, collaborative development, code management, and integration with data engineering workflows - 1 year of experience using AI for code generation, data analysis, automation, and enhancing productivity in data engineering workflows Nice to Have - 6 years of direct, hands-on experience performing business requirement analysis related to data manipulation, transformation, cleansing, and wrangling - 6 years of experience and strong technical knowledge of Microsoft SQL Server, including database design, optimization, and administration in enterprise environments - 2 years of experience extending or integrating data solutions with Azure Synapse Analytics and Microsoft Fabric, including Lakehouse, Warehouse, and Semantic Models - 1 year of direct experience building data products in a government owned cloud environment - 3 years of experience designing and integrating RESTful APIs, skilled in building secure, scalable RESTful APIs for data exchange with robust authentication, error handling, and support for real-time automation - 3 years of experience in message queueing technologies, implementing message queuing using tools such as ActiveMQ and Service Bus for scalable, asynchronous communication across distributed systems - 5 years of experience working with cross-functional teams to create software applications and data products