Data Engineer

Role: Data Engineer

Workplace Type: Remote (PST Hours)

Contract: Long Term

Job Details

The Data Engineer in the Data & AI Platform team is responsible for building, optimizing, and maintaining scalable data pipelines and infrastructure to enable secure, reliable, and high-performance access to enterprise data.

This role will focus on developing batch and near-real-time data ingestion pipelines, leveraging modern data platforms such as Databricks and Azure. The successful candidate will contribute to building high-quality, trusted data assets and analytics capabilities to support data-driven and AI-enabled decision-making across the enterprise.

Key Accountabilities

Build robust data pipelines: Design, develop, and maintain batch and near-real-time data pipelines for ingesting, transforming, and storing enterprise data.

Enable data-driven insights: Collaborate with analysts, data scientists, and business teams to ensure timely, reliable, and well-structured data is available to support analytics and AI use cases.

Develop near-real-time ingestion: Implement and optimize near-real-time data ingestion solutions into Databricks using appropriate streaming and integration technologies.

Support platform scalability: Contribute to the development of reusable data models, ETL frameworks, and metadata-driven pipelines across the enterprise data ecosystem, including Databricks, Azure Data Factory, and Power BI.

Implement data governance: Leverage Databricks Unity Catalog to manage data access, permissions, governance, lineage, and metadata.

Ensure data quality: Apply data validation, monitoring, and quality assurance techniques to maintain reliable and trusted data assets.

Monitoring and troubleshooting: Utilize Splunk for monitoring, log analysis, troubleshooting, and identifying issues across data pipelines and platform infrastructure.

Optimize data performance: Identify and implement performance improvements in data workflows, including partitioning, indexing, caching, and data format optimization.

Qualifications

Education & Experience

Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related technical field.

3–5 years of experience in data engineering or data platform development.

Hands-on experience building and maintaining data pipelines using Databricks and Apache Spark.

Experience implementing near-real-time data ingestion and processing solutions.

Experience working with enterprise data platforms and cloud-based data environments.

Technical Skills

Advanced proficiency in SQL and data pipeline development.

Strong hands-on experience with Databricks and Apache Spark (PySpark).

Practical experience with Databricks Unity Catalog, including data governance, access control, permissions, and data lineage.

Experience developing batch and near-real-time data ingestion pipelines into Databricks.

Experience with streaming technologies and data integration frameworks.

Hands-on experience with Splunk for log monitoring, analysis, troubleshooting, and operational visibility.

Experience with Azure Data Factory and other Azure data services.

Familiarity with cloud data platforms such as Azure Synapse or Snowflake.

Experience with data visualization tools such as Power BI or Tableau.

Strong understanding of ETL/ELT processes, data modeling, data quality, and data governance.

Core Competencies

Strong problem-solving and system design skills with the ability to build scalable and maintainable data solutions.

Effective communication skills with the ability to collaborate across technical and business teams.

Ability to translate business requirements into practical data engineering solutions.

Strong analytical and troubleshooting skills.

Demonstrated ability to thrive in a fast-paced, data-driven environment.

Ability to work independently and collaborate effectively within cross-functional teams.