Data Engineer - Databricks

Job Description – Data Engineer (AWS + Databricks)

Experience: 5+ Years

Location: Flexible / Hybrid

Key Responsibilities

• Build scalable ETL/ELT pipelines using Databricks (PySpark, Spark SQL, Delta Lake).
  

• Design and implement Medallion Architecture (Bronze, Silver, Gold) for Lakehouse.
  

• Develop batch and real-time pipelines using Spark Streaming, AWS Kinesis, or Glue Streaming.
  

• Manage data ingestion from diverse sources – RDBMS, APIs, S3, Kafka.
  

• Perform data profiling, validation, and quality checks.
  

• Optimize Spark jobs, partitions, and cluster configurations for performance.
  

• Implement and manage data infrastructure using AWS S3, Glue, Athena, Lambda, Step Functions, Redshift, EMR.
  

• Set up CI/CD pipelines (GitHub Actions, CodePipeline, Jenkins).
  

• Collaborate with analysts and data scientists to deliver analytical datasets.
  

Technical Skills

• Databricks: PySpark, Spark SQL, Delta Lake, Unity Catalog, Job Workflows.
  

• AWS: S3, Glue, Athena, Kinesis, Lambda, Step Functions, EMR, Redshift, CloudWatch.
  

• Programming: Python (pandas), SQL (window functions, optimization).
  

• IaC & DevOps: Terraform / CloudFormation, Git, Jenkins, GitHub Actions.
  

• Performance Tuning: Spark optimizations, Z-ordering, partitioning, caching.