Data Engineer Azure Databricks

Location: 100% Remote

Employment Type: Contract

Pay Rate: $45$55 per hour, based on experience

Work Authorization Requirement: Candidates must be legally authorized to work in the United States without current or future sponsorship. Additional eligibility requirements may apply based on the nature of the clients work.

Position Overview

A leading global aerospace and defense technology organization is seeking a Data Engineer to support its Enterprise Services Infrastructure team.

This role will focus on designing and maintaining enterprise data pipelines in Azure Databricks. The Data Engineer will integrate data from Microsoft 365, ServiceNow, Splunk, identity platforms, endpoint technologies, and cybersecurity systems into analytics-ready Delta Lake tables.

The position will also support data governance, lineage, semantic search, natural-language analytics, and applied AI initiatives. This is an excellent opportunity for a developing Data Engineer who wants hands-on experience with enterprise-scale data, modern Lakehouse architecture, and emerging AI technologies.

Key Responsibilities

Design, build, and maintain enterprise data pipelines using Azure Databricks.
Integrate ServiceNow, Splunk, Microsoft 365, identity, endpoint, infrastructure, and cybersecurity data.
Use Python, PySpark, and Spark SQL to transform raw data into analytics-ready Delta Lake tables.
Develop scalable ETL and ELT workflows using Delta Lake and medallion architecture.
Build Bronze, Silver, and Gold data layers to support analytics, reporting, and machine learning use cases.
Create and maintain data models that support Power BI reporting, operational analytics, and business decision-making.
Build automated data-quality checks, validation processes, monitoring, and error handling.
Develop data-lineage and governance solutions to identify upstream dependencies, downstream writes, undocumented transformations, stale datasets, ownership gaps, and redundant data products.
Help create semantic data catalogs that allow users to identify authoritative datasets and understand their business context.
Support natural-language analytics solutions that help users find relevant data and generate grounded analytical queries.
Contribute to retrieval-augmented generation solutions that connect current ServiceNow incidents with relevant historical resolutions.
Use vector embeddings, semantic search, metadata filtering, similarity matching, and clustering to improve enterprise knowledge management.
Help identify stale, duplicate, underused, missing, or low-quality knowledge articles.
Collaborate with infrastructure, identity, service desk, governance, cybersecurity, and analytics teams.
Translate operational challenges into scalable data products, reporting solutions, automation, and applied AI use cases.
Document pipelines, data models, transformations, dependencies, and engineering processes.

Required Qualifications

Approximately three or more years of hands-on data engineering experience.
Strong Python and SQL development skills.
Hands-on experience building data pipelines in Azure Databricks or a comparable Databricks environment.
Experience with PySpark, Spark SQL, Databricks notebooks, and Delta Lake.
Experience developing ETL or ELT pipelines and automated data workflows.
Understanding of medallion architecture and Bronze, Silver, and Gold data layers.
Experience with data modeling, data warehousing, schema design, and analytics-ready datasets.
Experience building data-quality validations, monitoring, and error-handling processes.
Ability to explain previous Databricks projects and individual technical contributions.
Must be legally authorized to work in the United States without current or future sponsorship and meet any additional client eligibility requirements.

Preferred Qualifications

Experience working with Microsoft Azure or Azure Government.
Experience integrating data from Microsoft 365, ServiceNow, Splunk, identity platforms, endpoint systems, or cybersecurity tools.
Experience with data governance, data lineage, metadata management, or semantic data catalogs.
Exposure to retrieval-augmented generation, vector databases, embeddings, or semantic search.
Experience with ChromaDB or another vector database.
Understanding of cosine similarity, clustering, metadata-filtered retrieval, or related matching techniques.
Exposure to large language model orchestration, prompt engineering, or structured LLM outputs.
Experience with Power BI, PostgreSQL, Git, or Docker.
Databricks Data Engineer or Machine Learning certification.
Bachelors or advanced degree in Data Engineering, Data Science, Computer Science, or a related field.

Why Join

Fully remote opportunity.
Competitive pay range of $45$55 per hour.
Work with Azure Databricks, PySpark, Spark SQL, Delta Lake, and medallion architecture.
Gain exposure to enterprise data from Microsoft 365, ServiceNow, Splunk, identity, endpoint, and cybersecurity platforms.
Contribute to both traditional data engineering and emerging applied AI initiatives.
Join a small, collaborative team with meaningful technical ownership.
Build data products that support technology adoption, operational decision-making, service improvement, and software ROI.
Expand your experience in data governance, semantic search, natural-language analytics, and retrieval-augmented generation.

Job Type & Location

This is a Contract position based out of New York, NY.

Pay and Benefits

The pay range for this position is $45.00 - $55.00/hr.

Individual compensation offered for this position within this range will depend on many factors, including qualifications, skills, relevant experience, job knowledge, geographic location, internal equity, and other pertinent job-related factors.

Eligibility requirements apply to some benefits and may depend on your job classification and length of employment. Benefits are subject to change and may be subject to specific elections, plan, or program terms. If eligible, the benefits available for this temporary role may include the following: Medical, dental & vision Critical Illness, Accident, and Hospital 401(k) Retirement Plan Pre-tax and Roth post-tax contributions available Life Insurance (Voluntary Life & AD&D for the employee and dependents) Short and long-term disability Health Spending Account (HSA) Transportation benefits Employee Assistance Program Time Off/Leave (PTO, Vacation or Sick Leave)

Job Type & Location

This is a Contract position based out of New York, NY.

Application Deadline

This position is anticipated to close on Oct 28, 2026.

Workplace Type

This is a fully remote position.

About TEKsystems

We're partners in transformation. We help clients activate ideas and solutions to take advantage of a new world of opportunity. We are a team of 80,000 strong, working with over 6,000 clients, including 80% of the Fortune 500, across North America, Europe and Asia. As an industry leader in Full-Stack Technology Services, Talent Services, and real-world application, we work with progressive leaders to drive change. That's the power of true partnership. TEKsystems is an Allegis Group company.

The company is an equal opportunity employer and will consider all applications without regards to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.

About TEKsystems and TEKsystems Global Services

Were a leading provider of business and technology services. We accelerate business transformation for our customers. Our expertise in strategy, design, execution and operations unlocks business value through a range of solutions. Were a team of 80,000 strong, working with over 6,000 customers, including 80% of the Fortune 500 across North America, Europe and Asia, who partner with us for our scale, full-stack capabilities and speed. Were strategic thinkers, hands-on collaborators, helping customers capitalize on change and master the momentum of technology. Were building tomorrow by delivering business outcomes and making positive impacts in our global communities. TEKsystems and TEKsystems Global Services are Allegis Group companies. Learn more at TEKsystems.com.

The company is an equal opportunity employer and will consider all applications without regard to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.

San Francisco Fair Chance Ordinance: Pursuant to the San Francisco Fair Chance Ordinance, for all positions located in the city and county of San Francisco, we will consider for employment qualified applicants with arrest and conviction records.

Massachusetts Lie Detector: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.

Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of our hiring process, including sourcing, screening, and evaluating candidates. AI helps assess applications and qualifications, but final decisions are made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools.

Data Engineer Azure Databricks