Software Engineer II (Data Engineering)

About the Role ZAGENO is hiring a Data Engineer to own the operational data pipelines powering our life sciences catalog. This is a high-ownership role: you’ll be the primary engineer responsible for the reliability, correctness, and scalability of the pipelines our operations teams depend on daily. The work spans production pipeline reliability, reducing accumulated technical debt in existing systems, building observability and testing infrastructure, and partnering with Data Science and Analytics on clean data delivery. You’ll work directly with CatalogOps and business stakeholders – turning evolving, often underspecified business rules into architectures that stay flexible without compromising data quality. You’ll make architectural decisions, push back on requests that introduce heuristic debt, and own incident response end-to-end. In this role you will: Own reliability and performance of operational pipelines across our product catalog infrastructure. Identify and reduce technical debt , replacing reactive patches with designed, testable logic. Build and maintain low latency data APIs that serve downstream operational and analytics consumers. Implement CDC patterns to keep catalog data synchronized across systems with minimal lag. Build monitoring, observability, and automated testing so failures surface before stakeholders report them. Design and implement unit standardization and master data logic at catalog scale. Translate business requirements from non-technical stakeholders into durable pipeline logic. Own code versioning, deployment, and incident response for your layer. Leverage your expertise in the tech stack: BigQuery, Databricks, Spark/PySpark, AWS/GCP. About you: Required: 3+ years as a Data Engineer, including solo or primary ownership of production pipelines Strong Python – data engineering, transformation logic, testing discipline Strong SQL with ability to write correct queries, identify and refactor anti-patterns Databricks, Delta Lake, Airflow for production orchestration Experience with CDC patterns for real-time or near-real-time data synchronization Experience building low latency APIs serving operational or analytical consumers Test-driven development discipline – unit tests, integration tests, regression coverage as standard practice, not afterthought Operates independently under ambiguity; designs systems to be maintained, not just to run Preferred: Kafka or equivalent event streaming platform experience Experience with entity matching, deduplication, or master data management Exposure to ML pipeline support in production Familiarity with NLP techniques for entity resolution or text normalization (tokenization, similarity matching, named entity recognition) What success looks like: Engineers ship data products: Outputs are documented, versioned, and designed for reuse across Analytics, Data Science, and operational consumers. Reliability: Pipelines run reliably with minimal manual intervention. Performance: Data latency and downtime decrease measurably over time. Data Quality: Analytics and Data Science teams receive clean, trustworthy data without ad hoc fixes. Scalability: Infrastructure scales with volume growth without proportional cost increase. Reduction of Debt: Technical debt in existing pipelines decreases measurably as heuristic patches are replaced with designed logic. Synchronization: CDC-driven synchronization eliminates the class of cross-pipeline identity drift bugs.