ES
Senior Ai Data Engineer Azure Databricks and Agentic Lakehouse
- Hiring from
- United Arab Emirates
- Work type
- Remote
- Posted
607,390 remote jobs, straight from company career pages
100% free · New jobs every minute
Show job descriptionHide job description
Job Description
Company: Efforts Solutions IT
Job Location: Remote Initially | Abu Dhabi, UAE (Potential Onsite Relocation)
Experience: 7+ Years
Reporting To: AI & Data Manager
About The Opportunity
Efforts Solutions IT is looking for a highly skilled Senior Data Engineer (Azure Databricks) to lead the architecture, development, and implementation of an enterprise-grade AI-powered Agentic Lakehouse for a major enterprise initiative.
This is a hands-on technical leadership role focused on building a modern data platform using Azure Databricks, Delta Lake, Unity Catalog, PySpark, and Azure Data Factory, integrating enterprise data with intelligent AI-driven automation.
The successful candidate will work closely with the AI & Data Manager, internal engineering teams, and external delivery partners to transform complex enterprise data into secure, governed, and reusable data products.
Work Arrangement & Relocation
About The Role
We are building an agentic lakehouse: a Databricks Medallion platform on Azure where AI agents help
capture, profile, clean and onboard enterprise data into governed, reusable data products. You will lead its
technical design and delivery, taking data from legacy operational systems to a trusted, well-governed
layer.
Key responsibilities
1 Lead the lakehouse architecture: ingestion, Bronze / Silver / Gold layering, Delta Lake standards and
data contracts.
2 Build ingestion pipelines from ERP databases, business applications, SharePoint and APIs using
Databricks and Azure Data Factory.
3 Set up Unity Catalog for governance: catalogs, lineage, access control and data classification.
4 Design AI agents that work on the data: schema inference, data-quality rule generation, anomaly
detection and entity resolution.
5 Expose governed data to AI agents through SQL endpoints, vector indexes and MCP-style data
services.
6 Embed data-quality checks with quarantine and human review for low-confidence fixes.
7 Set engineering standards and review the work of internal engineers and external partners.
8 Apply cost controls: cluster policies, compute choices and tagging.
Work scope
Azure Databricks
Role Level: Senior-Level Work Type: Full-Time Country: United Arab Emirates City: Abu Dhabi Company Website: https://effortz.com Job Function: Data Science & AI Company Industry/
Sector: Information Technology (IT)
What We Offer
Searching, interviewing and hiring are all part of the professional life. The TALENTMATE Portal idea is to fill and help professionals doing one of them by bringing together the requisites under One Roof. Whether you're hunting for your Next Job Opportunity or Looking for Potential Employers, we're here to lend you a Helping Hand.
Report
Disclaimer: talentmate.com is only a platform to bring jobseekers & employers together. Applicants are advised to research the bonafides of the prospective employer independently. We do NOT endorse any requests for money payments and strictly advice against sharing personal or bank related information. We also recommend you visit Security Advice for more information. If you suspect any fraud or malpractice, email us at abuse@talentmate.com.
Company: Efforts Solutions IT
Job Location: Remote Initially | Abu Dhabi, UAE (Potential Onsite Relocation)
Experience: 7+ Years
Reporting To: AI & Data Manager
About The Opportunity
Efforts Solutions IT is looking for a highly skilled Senior Data Engineer (Azure Databricks) to lead the architecture, development, and implementation of an enterprise-grade AI-powered Agentic Lakehouse for a major enterprise initiative.
This is a hands-on technical leadership role focused on building a modern data platform using Azure Databricks, Delta Lake, Unity Catalog, PySpark, and Azure Data Factory, integrating enterprise data with intelligent AI-driven automation.
The successful candidate will work closely with the AI & Data Manager, internal engineering teams, and external delivery partners to transform complex enterprise data into secure, governed, and reusable data products.
Work Arrangement & Relocation
- Initial Engagement: Remote for the first 2 months.
- Performance Evaluation: Technical capabilities and delivery performance will be assessed during the initial engagement.
- Potential Relocation: Following successful evaluation, the role may require relocation to Abu Dhabi, UAE, for an onsite or hybrid engagement.
- Contract Extension: Subject to project requirements, performance, and mutual agreement.
About The Role
We are building an agentic lakehouse: a Databricks Medallion platform on Azure where AI agents help
capture, profile, clean and onboard enterprise data into governed, reusable data products. You will lead its
technical design and delivery, taking data from legacy operational systems to a trusted, well-governed
layer.
Key responsibilities
1 Lead the lakehouse architecture: ingestion, Bronze / Silver / Gold layering, Delta Lake standards and
data contracts.
2 Build ingestion pipelines from ERP databases, business applications, SharePoint and APIs using
Databricks and Azure Data Factory.
3 Set up Unity Catalog for governance: catalogs, lineage, access control and data classification.
4 Design AI agents that work on the data: schema inference, data-quality rule generation, anomaly
detection and entity resolution.
5 Expose governed data to AI agents through SQL endpoints, vector indexes and MCP-style data
services.
6 Embed data-quality checks with quarantine and human review for low-confidence fixes.
7 Set engineering standards and review the work of internal engineers and external partners.
8 Apply cost controls: cluster policies, compute choices and tagging.
Work scope
- Assess current data sources, pipelines and data quality.
- Design and implement the lakehouse foundations: Medallion layers, Unity Catalog, ingestion
- Ingest and cleanse priority business domains from Bronze to Silver.
- Prototype AI agents for data profiling, quality rules and onboarding.
- 7+ years in data engineering, including leading lakehouse or data-platform builds.
- Proven track record defining target lakehouse architectures and data engineering standards.
- Deep hands-on Azure Databricks: Delta Lake, Unity Catalog (access models, lineage), DLT / Lakeflow,
- Strong PySpark and SQL; data modelling across Medallion layers.
- Hands-on experience building production ingestion and transformation pipelines with automated
- Azure data services: ADLS Gen2, Data Factory, Key Vault, private networking.
- Experience with data cleansing or migration for ERP systems.
- Practical use of LLMs in data workflows (data profiling, DQ rule generation, metadata, text-to-SQL).
- Git and CI/CD for data pipelines.
- Knowledge graphs or semantic layers.
- Databricks Data Engineer Professional certification.
Azure Databricks
- Unity Catalog
- Delta Lake
- ADLS Gen2
- Azure Data Factory
- Azure OpenAI
- Azure AI
- Azure DevOps
- Python / PySpark / SQL
Role Level: Senior-Level Work Type: Full-Time Country: United Arab Emirates City: Abu Dhabi Company Website: https://effortz.com Job Function: Data Science & AI Company Industry/
Sector: Information Technology (IT)
What We Offer
- Health Insurance
- Visa
Searching, interviewing and hiring are all part of the professional life. The TALENTMATE Portal idea is to fill and help professionals doing one of them by bringing together the requisites under One Roof. Whether you're hunting for your Next Job Opportunity or Looking for Potential Employers, we're here to lend you a Helping Hand.
Report
Disclaimer: talentmate.com is only a platform to bring jobseekers & employers together. Applicants are advised to research the bonafides of the prospective employer independently. We do NOT endorse any requests for money payments and strictly advice against sharing personal or bank related information. We also recommend you visit Security Advice for more information. If you suspect any fraud or malpractice, email us at abuse@talentmate.com.
