Data Engineer | Azure

Data Engineer | Azure join the team Make an impact by working for sectors where technology is the enabler, everything is ground-breaking and there’s a constant need to be innovative. Create and enhance projects in Java, Python, Angular, PHP, .NET and so much more while diving in the world of Blockchain, Artificial Intelligence, Data Science, Security and Internet of Things. Be part of the team that combines business knowledge, technological edge and a design experience. Our different backgrounds and know-how are key in developing solutions and experiences for digital clients. Face challenges and learn other ways of thinking and seeing the world - there’s always room for your energy and creativity. About the role Data Engineer is responsible for building and maintaining Data Platforms. Recognizes the importance of data for the organization in the areas where it is the key to success. Maintaining an eye on the big picture and knowing the details of the business are decisive for this role. This role focuses on designing, developing, and maintaining the data platform for data storage, processing, orchestration, and analysis. The mission involves implementing scalable, high-performance data pipelines and data integration solutions. Agnostic of data sources and technologies to ensure efficient data flow and high data quality, enabling data scientists, analysts, and other stakeholders to access and analyze data effectively. As a part of your job, you will: Design, build, and maintain scalable data platforms; Collect, process, and analyze large and complex data sets from various sources; Develop and implement data processing workflows using data processing framework technologies such as Spark and Apache Beam; Collaborate with cross-functional teams to ensure data accuracy and integrity; • Ensure data security and privacy through proper implementation of access controls and data encryption; Extraction of data from various sources, including databases, file systems, and APIs; Monitor system performance and optimize for high availability and scalability. What are we looking for? Technical Skills Profile: Databricks & PySpark Developer Databricks Ecosystem: Proficiency in building, managing, and optimising Databricks Workspaces and Notebooks . Experience with Databricks Unity Catalog for data governance, access control, and metadata management. Knowledge of Databricks Workflows (Jobs, Orchestration). Distributed Computing (PySpark): Strong hands-on experience using PySpark (Spark DataFrames, Datasets, and RDDs) for large-scale data processing. Deep understanding of Spark Architecture : Drivers, Executors, Workers, Partitions, Memory Management, and Shuffle operations. Ability to tune and optimise Spark applications (handling data skew, broadcast joins, caching strategies, and memory tuning). Languages & Data Warehousing Python: Familiarity with data analysis libraries (e.g., Pandas, NumPy) and standard library automation scripts. SQL: Expert-level SQL skills (complex JOINs, Window functions, CTEs, Aggregations, and Query Tuning). Experience with Spark SQL for data transformation and analytics. Data Architecture & Pipeline Engineering Data Pipelines & ETL/ELT: Designing, building, and maintaining high-volume batch and real-time streaming data pipelines (Structured Streaming). Implementation of Medallion Architecture (Bronze, Silver, and Gold layers) using Delta Lake. Incremental data loading patterns (Change Data Capture - CDC, MERGE INTO, Auto Loader). Data Warehousing Concepts: Knowledge of data modelling techniques (Dimensional Modelling, Star/Snowflake Schemas, Data Vault). Cloud Infrastructure, CI/CD & DevOps (Nice to Have / Preferred) Cloud Providers: Hands-on experience with Azure , integration with Databricks (e.g., AWS S3, Azure ADLS Gen2, Snowflake). DevOps & Software Engineering Practices: Version control using Git (GitHub, Azure DevOps, or GitLab). Experience with CI/CD pipelines for deploying Databricks code (using Databricks Asset Bundles, Terraform, or REST APIs). Automated testing for PySpark jobs (pytest) and data validation tools (e.g., Great Expectations) Personal traits Ability to adapt to different contexts, teams, and Clients; Teamwork skills but also a sense of autonomy; Motivation for international projects and ok if travel is included; Willingness to collaborate with other players; Strong communication skills. At Celfocus, we are committed to cultivate a diverse and inclusive workplace. As an equal-opportunity employer, we welcome applicants of all backgrounds, gender identities, and abilities. We are dedicated to providing reasonable accommodations for candidates with specific needs. If you require any adjustments during the selection process, please inform our Talent Acquisition Team. Come join the Team! Department BL D&A | Data Foundations | Data Engineering Role Data Engineer Locations Lisbon Work Model Hybrid Lisbon About Celfocus Celfocus is pioneering the future of business through Next-Gen Intelligence - the convergence of data, AI, and human creativity to build more autonomous and adaptive organisations. Guided by our core belief - Making Data Actionable - we partner with leading companies to co-create intelligent solutions that turn data into insight, agility, and business value. Operating in over 25 countries, we bring together business experts, data scientists, and AI engineers to turn complexity into clarity and vision into measurable impact. Founded in 2000, Celfocus is part of Novabase, listed on Euronext Lisbon. To know more, visit www.celfocus.com