Data Engineer Azure

Senior Azure Data Engineer / AI Platform Developer
Remote | Freelance / Independent Contractor | Project-Based Engagement
Project Overview
We are seeking an experienced Senior Data Engineer with strong Azure cloud, data engineering, and AI/ML platform development expertise to independently design, develop, and deliver a scalable enterprise-grade data platform.
The project involves building and improving data processing solutions, developing reliable data pipelines, integrating cloud-based services, and preparing data infrastructure to support AI, machine learning, and advanced analytics applications.
This is an independent contract engagement. The selected developer will take end-to-end responsibility for technical execution, from architecture and implementation through testing, documentation, and final delivery.
We are looking for someone who can work autonomously, make sound technical decisions, troubleshoot complex problems, and deliver production-ready solutions.
Key Responsibilities
1. Data Platform Architecture and Development
• Design and implement scalable Azure-based data engineering architectures.
• Develop enterprise-grade data platforms supporting structured and unstructured datasets.
• Build batch and real-time data processing solutions.
• Design efficient data storage, processing, and integration architectures.
• Identify opportunities to improve platform performance, reliability, scalability, and cost efficiency.
2. Data Engineering and ETL Development
• Build, maintain, and optimize end-to-end ETL/ELT pipelines.
• Develop data ingestion and transformation workflows.
• Implement data integration between internal systems, external APIs, and cloud services.
• Develop advanced SQL queries, stored procedures, database functions, and data models.
• Implement data validation, error handling, and data quality controls.
3. Azure Cloud and Distributed Processing
• Develop solutions using Azure Data Factory, Azure Functions, Event Hubs, and related cloud-native services.
• Implement event-driven architectures using Azure Event Grid and Service Bus.
• Work with Azure Databricks and distributed processing technologies.
• Optimize data workflows for high-volume and real-time processing.
• Support cloud migration, integration, and infrastructure improvements.
4. AI and Machine Learning Data Infrastructure
• Develop data processing frameworks supporting AI/ML applications.
• Prepare and transform datasets for machine learning and advanced analytics.
• Implement reliable data workflows throughout the AI/ML data lifecycle.
• Develop data algorithms and processing logic for analytical applications.
• Ensure data pipelines are scalable, consistent, and suitable for downstream AI/ML workloads.
5. Software Development and Integration
• Develop backend services and RESTful APIs.
• Integrate databases, cloud applications, and third-party systems.
• Implement secure and maintainable application components.
• Follow software engineering best practices, including version control, code reviews, and automated testing.
6. Deployment, Testing, and Documentation
• Configure and maintain CI/CD pipelines using Azure DevOps and GitHub.
• Perform unit, integration, and end-to-end testing.
• Identify and resolve technical issues independently.
• Produce technical documentation, architecture diagrams, and implementation guides.
• Provide a complete handover of the developed solution.
Required Technical Stack
Cloud and Infrastructure
• Microsoft Azure
• Azure Data Factory
• Azure Databricks
• Azure Functions
• Azure Event Hubs
• Azure Event Grid
• Azure Service Bus
• Azure Cosmos DB
• Azure SQL Data Warehouse / Azure SQL
Programming and Development
• Python
• SQL / T-SQL
• C# / .NET
• Java
• Node.js
• Unix Shell Scripting
Data Engineering and Processing
• ETL / ELT
• Apache Spark / PySpark
• Spark SQL
• Hive SQL
• Data Modeling
• RDBMS
• Structured and Unstructured Data Processing
• Batch and Stream Processing
• Event-Driven Architecture
DevOps and Integration
• Azure DevOps
• Git / GitHub
• CI/CD Pipelines
• RESTful APIs
• JSON / JSON-LD / XML
• Version Control and Automated Deployment
Candidates should have strong hands-on experience with Azure data engineering. Experience with AI/ML data infrastructure and enterprise-scale data processing is particularly important.
Expected Deliverables
The contractor will be responsible for producing the following:
1. Technical Architecture: A documented solution architecture, including system components, data flows, integration points, and infrastructure design.
2. Data Engineering Framework: A functional data processing framework capable of handling structured and unstructured data.
3. ETL/ELT Pipelines: Implemented and tested ingestion, transformation, and data integration pipelines.
4. Azure Integrations: Configured integrations across the required Azure services, databases, APIs, and event-driven components.
5. AI/ML Data Readiness: Data preparation and processing capabilities suitable for downstream AI, machine learning, and analytics applications.
6. Database Implementation: Optimized database structures, queries, stored procedures, and data models.
7. CI/CD Implementation: Working deployment pipelines, source control configuration, and automated build and release processes.
8. Testing and Quality Assurance: Test results covering functionality, data accuracy, performance, reliability, and error handling.
9. Technical Documentation: Complete documentation covering architecture, configuration, deployment, troubleshooting, and maintenance.
10. Final Handover: A fully functional implementation, source code, configuration files, deployment instructions, and knowledge-transfer documentation.
Required Qualifications
• 7+ years of professional experience in data engineering or software development.
• Strong hands-on experience designing and implementing Azure cloud solutions.
• Extensive experience with relational databases and advanced SQL.
• Demonstrated experience developing production-grade data pipelines.
• Experience with distributed data processing and large-scale datasets.
• Practical understanding of AI/ML data engineering requirements.
• Experience implementing cloud integrations and event-driven architectures.
• Strong understanding of Git, CI/CD, and automated deployment.
• Ability to independently manage technical work from initial design through final delivery.
• Excellent debugging, analytical, and problem-solving skills.
Engagement Expectations
• Work Arrangement: Fully remote.
• Engagement Type: Freelance / Independent Contractor.
• Project Ownership: End-to-end technical responsibility.
• Working Style: Independent execution with milestone-based progress reporting.
• Communication: Regular updates, documented progress, and timely identification of blockers.
• Code Ownership: All project-specific source code, technical documentation, and agreed deliverables must be handed over to the client.
• Confidentiality: All project information, credentials, datasets, and technical materials must be handled securely.
Ideal Candidate
The ideal candidate is a senior-level Azure Data Engineer who combines strong software development capabilities with practical experience building enterprise data platforms.
We are particularly interested in professionals who can operate independently, translate technical requirements into working solutions, take ownership of complex engineering tasks, and deliver maintainable systems without requiring constant supervision.
If you have the technical expertise and confidence to independently own and deliver a complex Azure data engineering and AI platform project, we would like to hear from you.
Please include your relevant project experience, technical expertise, portfolio or GitHub profile, availability, and proposed engagement rate when applying.