Senior Data Engineer
- Hiring from
- Probably Worldwide
- Work type
- Remote
- Posted
Is this job info correct?
513,132 remote jobs, straight from company career pages
100% free · New jobs every hour
Show job descriptionHide job description
Key Responsibility Areas:
- Design and develop scalable, secure, and high-performance data engineering solutions on Microsoft Azure.
- Architect and implement complex ETL/ELT data pipelines using Azure Data Factory and Azure Databricks.
- Develop highly optimized PySpark applications and Databricks notebooks for large-scale data processing.
- Design and implement batch and incremental data processing frameworks.
- Develop complex SQL queries, stored procedures, CTEs, window functions, and data transformations.
- Design and implement data ingestion solutions from multiple structured and semi-structured data sources.
- Work extensively with Azure Data Lake Storage Gen2, Delta Lake, and Parquet.
- Design and implement Medallion Architecture (Bronze, Silver, Gold) and modern data lake/lakehouse solutions.
- Optimize Databricks clusters, Spark jobs, PySpark code, SQL queries, and data pipelines for performance and cost.
- Develop robust error handling, logging, monitoring, and data validation mechanisms.
- Implement data quality, reconciliation, and validation frameworks.
- Troubleshoot complex production issues and provide root-cause analysis and permanent solutions.
- Lead technical discussions and contribute to data architecture and solution design.
- Mentor junior and mid-level data engineers and provide technical guidance.
- Collaborate with Data Architects, Developers, Data Scientists, Business Analysts, and other stakeholders.
- Participate in code reviews and establish engineering best practices and coding standards.
- Implement CI/CD processes for data engineering workloads using Azure DevOps/Git.
- Ensure solutions comply with organizational standards for security, governance, scalability, and reliability.
Mandatory Technical Skills
- 7–11 years of overall experience in Data Engineering / Data Platforms.
- Strong hands-on experience with Azure Databricks.
- Strong experience with Azure Data Factory (ADF).
- Excellent hands-on experience with PySpark and Python.
- Advanced proficiency in SQL.
- Strong understanding of Apache Spark and distributed data processing.
- Experience with Delta Lake and Parquet.
- Strong experience designing and developing ETL/ELT pipelines.
- Strong understanding of data warehousing and data modeling concepts.
- Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
- Experience with performance tuning and optimization of Spark/PySpark workloads.
- Experience with production support, troubleshooting, monitoring, and data pipeline optimization.