Data Engineer

Job Description This is a remote position. About Verizol Verizol is ZipNom's company intelligence and verification platform: 45+ APIs (KYC, KYB, bank, face, OCR, CKYC, eSign) built on a daily MCA data pipeline, sold through a single API key and prepaid credit wallet. Customers are CA firms, fintechs, NBFCs, and marketplaces. The engineering blueprint is written, the 8-week build plan is set, and the team is small enough that your name is on what ships. The Role You own the asset the entire company is built on. Every Verizol product — free search, daily alerts, KYB reports, watchlists — reads from one normalized, indexed database of Indian company registrations, and that database exists because your pipeline ingests, parses, and diffs MCA data every single day. This is the least substitutable seat on the team: when a government data source silently changes format at 2 AM, you're the one whose alarms catch it and whose parsers get fixed before customers notice. This is not clean-API ETL. It's real-world data engineering against uncooperative sources — scraping, format archaeology, checksum validation, and building a system honest enough to know when its own data is stale. Requirements What You Will Do Build source ingestors for MCA data (new incorporations, master data, index of charges, DIN registry, filing metadata) with raw-first storage to S3 — every byte stored before parsing, so parser bugs are always replayable Write parsers as pure, fixture-tested functions: raw bytes in, typed rows out, with a fixture for every format variant ever observed in the wild Build and own the snapshot + diff engine: detect exactly what changed for every company daily, and emit granular change events (director resigned, charge created, status changed) — the event stream that powers Alerts and Watchlists Own the daily production schedule: ingest 02:00 → diff 04:00 → digests 06:30 → alert emails at 08:00 IST, with stage-level retries and a hard rule that stale data never silently ships Build the alerts digest job: filter matching across all subscriber streams, email rendering, SES delivery — for CA subscribers, this email is the product Build the monitoring that keeps you sane: row-count anomaly alarms, parser error rates, per-source freshness gauges Maintain the reference tables (state/ROC mappings, NIC codes) and run the historical backfill (24+ months of incorporations) Required Skills 3–6 years of data engineering or backend work with production scraping/parsing experience against uncooperative sources — this is the non-negotiable; clean-API ETL alone won't prepare you for MCA Strong Python: Celery (or equivalent task queues), lxml/BeautifulSoup, pandas, pdfplumber or similar document parsing Solid PostgreSQL: bulk upserts, COPY, partitioning, and index design for time-series feed queries Testing instincts for data: fixtures, golden files, replay tests — you can prove a parser change didn't corrupt yesterday Operational maturity: your jobs run while everyone sleeps; you build the alarms first and take the 2 AM page seriously Comfort with proxies, rate budgets, and being a polite, resilient client of fragile infrastructure Preferred Prior experience with Indian government/registry data: MCA21, GSTN, EPFO, court records, or similar Familiarity with CIN/DIN structures, ROC organization, or corporate filings AWS: S3 lifecycle policies, SQS, CloudWatch metrics/alarms Email deliverability basics (SES, DKIM) — you'll co-own the digest send