Senior Data Engineer Databricks, PySpark

Build the Future with AspenView Technology Partners At AspenView, we are passionate about transforming the way organizations approach technology. We specialize in creating high-performing, nearshore IT teams to help North American clients innovate faster and more efficiently. As we continue to grow, we’re looking for exceptional people to join our team and help drive impactful change across industries. Why Join AspenView? At AspenView, we’re more than a nearshore IT partner—we’re a people-first, purpose-driven company that believes great culture drives great outcomes. We’re passionate about connecting talent and technology to deliver measurable value for clients—and meaningful career paths for our people. Here’s what you can expect: Competitive base Flexible work model: hybrid, remote, or in-office Real growth opportunities and leadership visibility Inclusive, respectful culture that blends U.S. innovation with Colombian heart A company that listens, invests in you, and celebrates wins together We are looking for a skilled Senior Data Engineer with hands-on experience in Databricks, Unity Catalog, and Azure Data Factory (ADF) to support a large-scale enterprise data modernization initiative. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, enabling secure data governance, and supporting cloud-based analytics solutions in Azure. What you will do: This role will collaborate closely with architects, business stakeholders, and cross-functional engineering teams to build reliable, high performance data platforms and accelerate data-driven decision-making. AI Application Development & Orchestration Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks. Build and maintain scalable data ingestion frameworks for structured and unstructured data sources. Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks. Implement and manage Unity Catalog for enterprise data governance, security, and access controls. Create and optimize Delta Lake architectures for analytics and reporting workloads. Monitor pipeline performance and troubleshoot production issues. Collaborate with data architects to implement enterprise data model and best practices. Ensure data quality, integrity, security, and compliance standards are met. Support CI/CD deployment processes and DevOps practices for data engineering solutions. Work closely with business and analytics teams to understand requirements and deliver scalable solutions. What you bring: Experience: 4-7+ years of experience in Data Engineering or Big Data development. Strong hands-on experience with Azure Databricks. Proficiency in PySpark, Spark SQL, and distributed data processing. Experience implementing and managing Unity Catalog. Strong experience with Azure Data Factory (ADF). Experience working with Delta Lake and modern Lakehouse architectures. Solid understanding of data modeling, ETL/ELT, and data warehousing concepts. Experience with Azure cloud services and ecosystem. Proficiency with Git, version control, and CI/CD processes. Strong analytical and problem-solving skills. Excellent communication and stakeholder management abilities. Preferred Qualifications: Experience with Azure Synapse Analytics. Familiarity with Collibra or other data governance platforms. Experience with Infrastructure as Code (Terraform, ARM Templates). Databricks Certification (Data Engineer Associate/Professional). Microsoft Azure Data Engineer Associate Certification Nice to have: Experience supporting enterprise-scale data migration projects. Knowledge of healthcare, retail, or consumer products data environments. Exposure to real-time data processing and streaming technologies. Visa Sponsorship AspenView does not sponsor employment visas for this role. Applicants must be currently authorized to work in the United States on a permanent basis without the need for visa sponsorship now or in the future. Equal Opportunity Employer AspenView is proud to be an equal opportunity employer. We believe in creating an environment where all employees feel welcome, valued, and empowered to succeed. We celebrate diversity and strive to build a culture of inclusion where all individuals, regardless of their race, color, gender, gender identity or expression, sexual orientation, disability, age, or any other characteristic, can thrive. We encourage applicants from all walks of life to join our team and make a lasting impact.