Sr. Data Engineer - Flink

Job Description This is a remote position. Building high-scale, low-latency streaming data pipelines deployed on infrastructure we run ourselves (on-prem), not managed cloud services. You will design and operate high-volume real-time data systems end to end, with Apache Flink as the core stream-processing engine. Requirements • 7+ years of experience in data engineering and software development • Ability to write high-quality code in Java/Scala, Python, or equivalent languages • Deep, hands-on production experience with Apache Flink — DataStream API and Table API / Flink SQL (core requirement) • Demonstrated experience with Flink state management: keyed state, state backends (e.g., RocksDB), large state sizes, and state TTL • Hands-on experience with checkpointing, savepoints, and fault tolerance — exactly-once vs. at-least-once semantics, recovery, and savepoint-based job upgrades • Strong grasp of event-time processing: watermarking, windowing strategies, allowed lateness, and late-data handling • Experience diagnosing and resolving backpressure — parallelism, operator chaining, and network buffer tuning • Experience operating Flink on self-managed infrastructure (Kubernetes or YARN) — application vs. session mode, high availability, and rolling upgrades • Practical experience with stream processing (Kafka Streams or equivalent) and messaging systems for high-volume workloads, including exactly-once sinks and schema registry usage • Practical experience with distributed query engines (e.g., Trino/Presto or similar) • Practical experience with ETL / data integration tools, commercial or open-source (e.g., Datastage, Informatica, Apache NiFi, or similar) • Practical experience with SQL-based transformation frameworks (e.g., dbt or others) • Strong SQL skills and understanding of data modeling and data warehousing for analytical workloads • Hands-on experience with real-time / low-latency analytical stores (columnar or OLAP engines, e.g., Apache Pinot/ClickHouse or similar) • Practical experience with big-data platforms and distributions (e.g., Cloudera, Hadoop ecosystem, or similar) • Practical experience containerizing and operating data workloads (Docker; Kubernetes a plus) • Experience with workflow orchestration tools (e.g., Airflow or similar) • Familiarity with data lake table formats (e.g., Apache Iceberg or similar), including streaming ingestion, compaction, and small-file management • Familiarity with data governance / cataloging tools (e.g., DataHub or similar) • Familiarity with lakehouse management systems (e.g., Apache Amoro or similar) • Familiarity using AI tools for development and debugging (Claude, Cursor, Codex)