Senior Data Engineer

Seamless delivers the world’s best sales leads. Through our product, we help sales teams maximize revenue, increase sales, and easily acquire their total addressable market using artificial intelligence; by development of a robust real-time contact and company search engine as well as a suite of technically-advanced tools to support sales and lead generation. We have been recognized as one of Ohio’s fastest growing companies and have been recently ranked No. 7 in LinkedIn's Top 50 Startups of 2022, featured in Forbes as #1 Software company in Ohio in 2022, and on G2’s “Top 100 Highest Satisfaction Products for 2022” list! We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status. Visa Sponsorship is not included in our hiring package. Applicants must be authorized to work in the U.S. The Opportunity The Senior Data Engineer will play a critical role in our expanding Data Product Team. They will work hands-on with our entire company and contact profile universe with the main objective of improving the coverage, accuracy, and infrastructure scalability of our product data. This person will be tasked with solving significant data problems that are positively impacting millions of business professionals in the ever evolving lead intelligence space. The Senior Data Engineer will have the opportunity to work with cutting edge technology in big-data, ETL orchestration, data analytics, machine learning, and AI. About the Role AWS-Based ETL Pipeline - Deep Data Engineering - Infrastructure Performance - Data Analytics This is a hybrid data engineering and analytics role, not a pure pipeline-building position. Much of the work involves investigating data-quality issues, analyzing why current scoring rules produce bad outcomes, and deciding how they should change — that requires real analytical and statistical reasoning on top of the engineering work, not just building infrastructure to move data from A to B. You'll be expected to own architecture decisions independently and work with limited day-to-day guidance. Our pipeline processes billions of records , so this is not a role for someone who has only worked with SQL/Spark at small-to-moderate scale. Query and job design choices here have real cost and runtime consequences, and inefficient code fails or times out in ways it wouldn't on a smaller dataset. What You'll Work On Building and maintaining PySpark ETL jobs Supporting ML-based scoring models (e.g., logistic regression classifiers) that predict the quality/validity of contact data (phone numbers, emails) Writing and optimizing SQL used in data preparation steps (joins across large "silver/gold" lake tables) Collaborating on architecture decisions for scoring systems (e.g., tiered status gating + ML-based fine positioning within a score range) Writing unit and integration tests for data transformations and model inference code Working within a medallion-style lake architecture (bronze/silver/gold layers) Identify opportunities for leveraging AI and agentic workflows for data acquisition, identity resolution, and profile aggregation Required Skills Experience Level 5+ years of professional data engineering experience, with demonstrated ownership of projects end-to-end (design through production) Comfortable making and defending architecture decisions with minimal oversight Comfortable owning a project end-to-end: from analysis/planning through implementation and testing Able to work with dependencies on other teams' data deliverables and adjust scope accordingly Clear communicator who can document architecture decisions and tradeoffs Data Engineering Strong Python skills, including pyspark dataframes Deep, hands-on experience with Apache Spark / PySpark, including how Spark actually executes a job — partitioning, shuffles, skew, broadcast joins, predicate/partition pruning, caching, and reading a physical query plan ( explain() ) to find and fix bottlenecks Complex SQL fluency is required : multi-way joins, window functions, CTEs, and aggregations written to perform well at billion-row scale, not just to produce a correct result — this pipeline runs against billions of records, and naively-written SQL that's fine on a small dataset will fail or time out here Experience with AWS data services: Glue (jobs/crawlers), EMR, S3, Step Functions, Lambda, EventBridge, CloudWatch Experience writing automated tests for data pipelines (pytest or similar) Data Flow & Systems Understanding Demonstrated ability to trace data lineage across a multi-stage pipeline (e.g., bronze/silver/gold) rather than reasoning about a single job in isolation Real orchestration experience (Step Functions, Airflow, or similar) with DAG-style, fan-out/fan-in dependency chains — not just standalone scripts Experience modeling and deduplicating records from many heterogeneous upstream sources (e.g., multiple CRMs, public data feeds) into a canonical schema Able to debug across distributed AWS services by following an execution trail (e.g., EventBridge → Lambda → Glue → Step Functions) using CloudWatch logs Documents what they learn as they trace a data flow, so institutional knowledge doesn't stay siloed in one person's head Analytical / Data Quality Reasoning Exploratory data analysis at scale — comfortable using SQL to explore distributions and segment by source or bracket to see where current rules break down, before writing any pipeline code Ground-truth/label skepticism — recognizing that signals like disposition data are noisy proxies, not ground truth, and reasoning about how that noise propagates into score quality Classification statistics — precision/recall, false-positive vs. false-negative cost tradeoffs, calibration — to judge whether a scoring change actually improves quality or just shifts errors around Backtesting / historical validation — testing a proposed scoring rule change against historical outcomes before shipping it, rather than shipping and hoping Root-cause analysis on tickets — tracing a reported data-quality complaint back to the specific rule, feature, or upstream source causing it Translating ambiguous goals into concrete criteria — turning something like "improve phone accuracy" into specific, testable threshold/feature decisions Data Validation & Verification Discipline Reconciles row counts and cardinality at each pipeline stage, and understands why counts should (or shouldn't) match — e.g., catching a join that silently fans out 1:many instead of 1:1 Diffs output before and after a change as a default habit, not an afterthought, to see exactly what shifted and whether that shift was intended Profiles data on both input and output (null rates, distinct counts, min/max, distribution shape) rather than trusting a job that completed without exceptions Recognizes silent failure modes — duplicate keys, null-key drops, schema drift, type coercion, encoding issues — that produce wrong data without ever throwing an error Writes or uses automated data quality checks (assertions, dbt-style tests, Great Expectations, custom row-count/null-rate checks) so validation doesn't depend on someone remembering to look Follows a change through to confirmed-correct output, not just to "deployed successfully" Nice to Have Practical experience building and deploying classical ML models (e.g., logistic regression, gradient boosting) and feature engineering from tabular, multi-source data — this is not a deep learning role, and not required, but directly applicable to this project Experience with scikit-learn or comparable ML libraries, and turning a trained model into a production inference class/service Experience with contact data quality, identity resolution, or marketing/sales data enrichment Familiarity with CI/CD for data pipelines (this repo uses GitHub Actions) Experience with EMR or other managed Spark environments Prior work on scoring systems that blend rule-based gating with ML-based ranking Experience successfully developing and managing LLM’s and agentic workflows