Sr. Data Engineer

About the Job

As a Data Engineer at Aspire, you will work closely with the data engineering and data platform teams to support a large-scale data platform modernization initiative. You will play a key role in migrating approximately 1,000 datasets to Snowflake, modernizing existing data pipelines, and improving data processing reliability and performance.

You will focus on executing established technical migration plans, adapting existing pipelines, and supporting orchestration using Apache Airflow 3. This role offers the opportunity to contribute to critical data platform initiatives, including Airflow upgrades, Confluent regional migrations, and the transition from Hive to Apache Iceberg, with a target completion date of March 2027.

What you’ll do

• Migrate existing data pipelines from Apache Airflow 2 to Apache Airflow 3.
  
• Migrate Confluent connections from the USE1 region to the USE2 region.
  
• Migrate data pipelines from Hive to Apache Iceberg using Snowpark Connect.
  
• Adapt and migrate existing data pipelines to Snowflake as part of a large-scale data platform modernization effort.
  
• Develop, test, and maintain data pipelines using Python, SQL, Apache Airflow, and Apache Spark.
  
• Ensure data quality, accuracy, reliability, and consistency throughout the migration process.
  
• Support workflow scheduling, monitoring, troubleshooting, and performance optimization in Apache Airflow.
  
• Work with large-scale datasets and AWS services, particularly Amazon S3, to support data lake operations and data processing.
  
• Implement testing and debugging strategies to identify and resolve issues across data pipelines and processing workflows.
  
• Collaborate with data engineering and data platform teams to execute migration plans and meet project deadlines.
  
• Contribute to the successful delivery of time-sensitive platform modernization initiatives targeted for completion by the end of March 2027.
  

What you’ll need

• 5+ years of experience in data engineering or software engineering.
  
• Strong software engineering fundamentals and hands-on Python development experience.
  
• Advanced SQL skills, including complex joins, common table expressions (CTEs), window functions, and query performance tuning.
  
• Experience with Snowflake data modeling and data warehouse concepts.
  
• Hands-on experience developing, testing, and maintaining data pipelines using Apache Airflow and Python.
  
• Experience working with Apache Kafka and distributed data streaming platforms.
  
• Strong experience with Apache Spark, including hands-on PySpark development.
  
• Experience with Apache Iceberg and Parquet data formats.
  
• AWS experience, particularly Amazon S3, including data lakes, partitioning strategies, and lifecycle policies.
  
• At least 2 years of industry experience working with large-scale datasets using technologies such as Apache Airflow and Apache Spark.
  
• Strong testing, debugging, and troubleshooting skills, with the ability to investigate and resolve issues independently.
  
• Excellent analytical, problem-solving, and communication skills.
  
• Ability to work independently, adapt to established technical architectures, and deliver results within tight deadlines.
  
• Strong collaboration skills and the ability to work effectively with distributed engineering teams.
  
• Experience with IT security best practices and standards such as ISO or SOC.
  
• Hands-on experience with Snowflake migrations, including data warehouse modernization, performance optimization, and cost optimization.
  
• Experience migrating Apache Airflow environments, including version upgrades, executor changes, or platform migrations.
  
• Experience with Confluent and Kafka infrastructure migrations across regions.
  
• Experience migrating data pipelines from Hive to Apache Iceberg.
  
• Experience using AI-powered tools to support software development, testing, and debugging.
  

Why Aspire

• Be part of a remote-first organization where flexibility is embraced.
  
• Work and learn alongside talented engineers and technology leaders.
  
• Explore opportunities to learn and grow through technical and non-technical training programs.
  
• Gain global exposure by working on products with international teams and clients.
  
• Attend virtual and in-person international technology conferences to expand your knowledge and network.
  
• Exposure to working in an IT environment that adheres to rigorous security and compliance standards defined by ISO / SOC.