REMOTE - Full-Stack Data Engineer (Databricks, FastAPI & Next.js)

Company Description Celebal Technologies is a software services company specializing in Data Science, Big Data, and Enterprise Cloud solutions for global clients. The company focuses on turning “Data to Intelligence” by using advanced analytics, AI, ML, and robotics to drive smarter, faster decision-making and improved business efficiency. Celebal Technologies partners with product ISVs and enterprise delivery organizations to support product development and large-scale implementations across industries. The team has strong expertise in modern and emerging technologies such as Blockchain, IoT, analytics and visualization, and cloud-native services, helping organizations modernize legacy systems and become truly data-driven. Its tailored solutions are designed to maximize productivity, enhance speed and accuracy, and elevate business intelligence.

Job Title: Full-Stack Data Engineer (Databricks, FastAPI & Next.js)

Location: REMOTE or Hybrid from Brisbane QLD 4000 (For Local Candidates)

Duration: 3+ Months Initial (Extendable up to 6+ months)

Experience: 3–5 years of software development experience, with hands-on expertise in Python, Databricks, and Next.js/React.

Job Summary

We are seeking a highly motivated Full Stack Data Engineer to build modern, scalable, cloud-native applications that leverage Databricks for data processing, Python for backend services and APIs, and Next.js for responsive web applications.

The ideal candidate will have experience developing enterprise-grade applications that integrate with analytics platforms, data pipelines, AI/ML services, and cloud-native technologies. The role requires strong backend engineering skills, modern frontend development capabilities, and experience working with Databricks as the compute platform.

Key Responsibilities

Databricks & Compute

• Develop and maintain data processing workloads using Databricks.
• Build PySpark notebooks and workflows for data transformation and analytics.
• Integrate applications with Databricks Jobs, Workflows, SQL Warehouses, and REST APIs.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Work with Delta Lake, Unity Catalog, and Medallion Architecture.
• Develop data APIs and services that consume Databricks datasets.

Backend Development (Python)

• Design and develop scalable backend services using Python (FastAPI, Flask, or Django).
• Build RESTful APIs and microservices.
• Develop authentication and authorization mechanisms (OAuth2, JWT, SSO).
• Integrate backend services with Databricks, Azure services, and third-party APIs.
• Implement asynchronous processing and background jobs.
• Write reusable, well-tested, and maintainable code.
• Build logging, monitoring, and error-handling capabilities.

Frontend Development (Next.js)

• Develop responsive web applications using Next.js and React.
• Create reusable UI components and modern dashboards.
• Integrate frontend applications with backend APIs.
• Implement authentication and role-based access control.
• Optimize application performance and user experience.
• Ensure responsive design across desktop and mobile devices.

Integration & Cloud

• Integrate applications with Azure cloud services.
• Consume APIs from enterprise applications and external systems.
• Implement secure API communication and data exchange.
• Participate in cloud deployment and infrastructure discussions.

DevOps & CI/CD

• Use Git for source control and collaborative development.
• Build and maintain CI/CD pipelines using Azure DevOps or GitHub Actions.
• Support automated testing and deployment processes.
• Participate in code reviews and enforce coding standards.

Quality & Security

• Develop unit and integration tests.
• Ensure applications meet enterprise security and compliance requirements.
• Implement secure coding practices and vulnerability remediation.
• Troubleshoot production issues and support release activities.

Must Have:

Data Engineering core concepts

• Databricks
• ETL Pipelines, Medallion Architecture, PySpark
• Databricks Automation Bundles
• 
  
• Python Backend
• FastAPI (Must)
• REST API Development
• AuthN Basics (JWT, OAuth2)
• Databricks App builder exp including deployment
• 
  
• General Engineering
• Agile
• Unit Testing
• CI/CD
• Azure DevOps

Good to Have:

• Front end design/development
• Next.js or similar
• Html/css framework
• AI Engineering
• ML Engineering
• MLOps

ADDITIONAL REQUIREMENTS

• Strong communication capabilities for effective stakeholder coordination.

• Immediate local availability within Brisbane or complete openness to travel onsite as needed per project milestones.

• Unrestricted working rights in Australia.