AI Data Engineer

🚀 Python / AI Data Engineer – Web Scraping, LLM & Databricks 🚀

We are looking for an experienced Python / AI Data Engineer to design and develop an automated solution for collecting, extracting, and structuring technical product and asset data from publicly available web sources.

The ideal candidate has strong hands-on experience with Python, Web Scraping, Large Language Models (LLMs), and Databricks, along with expertise in data extraction, processing, and automation.

📍 100% Remote | 280 Hours

Key Responsibilities

• Design and develop scalable web crawling and scraping solutions to collect technical data from websites, product catalogs, and online databases.
• Build automated data extraction pipelines using Python, Scrapy, BeautifulSoup, or similar frameworks.
• Implement AI-powered information extraction using LLMs, NLP, and prompt engineering.
• Extract and process structured and unstructured data from HTML pages, PDFs, technical documents, and APIs.
• Develop data validation, normalization, classification, and duplicate detection processes.
• Transform extracted information into structured datasets and standardized CSV files.
• Develop a Databricks App for data quality checks, validation, and review.
• Implement data processing workflows using Pandas, PySpark, and Delta Lake.
• Conduct testing, optimize extraction accuracy, and ensure data quality.
• Prepare technical documentation and support solution deployment and handover.

Requirements

• Strong hands-on experience with Python and data engineering.
• Proven experience in Web Scraping and Web Crawling using Scrapy, BeautifulSoup, or similar frameworks.
• Practical experience with LLMs, NLP, Generative AI, and Prompt Engineering.
• Experience with structured data extraction, classification, normalization, and data validation.
• Strong knowledge of Databricks, PySpark, Pandas, and Delta Lake.
• Experience developing Databricks Apps.
• Experience working with REST APIs, PDF extraction, and document processing.
• Good understanding of HTML, DOM structures, and dynamic websites.
• Strong analytical, troubleshooting, and problem-solving skills.
• Ability to work independently and deliver end-to-end technical solutions.

Nice to Have

• Experience with Selenium, Playwright, or browser automation tools.
• Knowledge of Azure services, MLOps, and CI/CD pipelines.
• Experience with product catalogs, asset data, or technical data models.
• Familiarity with SAP EAM or enterprise asset management systems.
• Understanding of electrical engineering terminology and technical specifications.

Project Details

• Location: 100% Remote
• Project Duration: November – December 2026
• Workload: 280 hours total (approximately 35 hours/week)
• Eligibility: EU citizenship required

Interested in this opportunity? We'd be happy to hear from you!

📩 Apply by sending your CV to Career@crewharbor.de

#Hiring #PythonDeveloper #AIEngineer #DataEngineer #WebScraping #Databricks #LLM #GenerativeAI #PySpark #RemoteJobs #B2B #CrewHarbor