Principal Backend / Performance / Database Engineer

Context

iClosed.io is a SaaS platform for sales and marketing teams, and we're growing quickly. As usage scales to millions, performance becomes a core part of the product experience, so we're investing in it early and deliberately. We've formed a dedicated Performance Team to lead that work.

We run a polyglot data layer spanning relational, document, key-value and in-memory stores, and a backend built across multiple modern languages and frameworks. We believe the biggest gains come from design rather than tuning alone: matching each store to its access pattern, moving work off the critical path, and drawing service boundaries that scale cleanly. That's why this role is weighted toward system design. We want someone who solves the problem in front of them, then shapes the architecture so entire classes of problems never come up.

The role

Title: Principal Backend / Performance / Database Engineer
Reports to: CTO / Product Manager
Type: Full-time, permanent
Level: Senior individual contributor. No direct reports, high autonomy, significant technical influence.

You own the performance and scalability of our backend systems and the data layer underneath them. The work splits roughly into three parts:

• Design: how a request flows, what's synchronous, where state lives, and what happens when a dependency is slow.
• Diagnosis: reading traces, plans and metrics to find the real cause rather than the obvious one.
• Prevention: instrumentation, review standards and regression gates, so problems surface before customers notice.

We want genuine depth in both halves: someone who can design a system that scales, and who can also open an execution plan or a key-value access pattern and see what's wrong with it. Candidates strong in only one half will find this role frustrating.

What you'll own

System and service design

You'll recognise which of these a problem calls for, and what each costs:

• Scaling reads: caching, read replicas and index strategy, with invalidation and staleness decided deliberately.
• Scaling writes: sharding, partitioning, batching, and queue-buffered writes where needed.
• Read/write separation: replica routing, and CQRS where it earns its complexity.
• Background processing: queues, workers, idempotency, retries, dead letters and backpressure.
• Real-time updates: WebSockets, SSE or polling, and their fan-out and cost implications.
• Traffic spikes and high availability: autoscaling on leading indicators, headroom by choice, failover, and a clear answer to what degrades when a component is lost.
• Reliability: timeouts, backoff, circuit breakers and graceful degradation.
• Search, data growth and safe change: dedicated search indexes, archival and retention, feature flags, and online migrations that don't lock production.
• API design: pagination, rate limits, payload budgets, and batch rather than chatty access.

You'll also review feature teams' architecture proposals, and write the design docs and decision records other engineers build from.

Data layer across engines

Deciding which workload belongs where, and correcting cases where that was decided by habit:

• Relational: execution plans, index write cost, lock contention, connection pooling, replica lag and partitioning.
• Document: modelling, index and shard key design, aggregation performance and working set sizing.
• Key-value: access-pattern-first modelling, hot partition avoidance, secondary index strategy and capacity cost.
• In-memory cache: caching patterns, key design, eviction, memory pressure and cold-cache behaviour.
• Across engines: consistency boundaries and dual-write problems.

Latency and throughput

• Diagnose latency across the full request path with evidence.
• Eliminate N+1 and chatty access, including ORM-generated traffic.
• Reduce sync and ETL latency, including third-party integration paths.
• Own capacity planning for the data layer, and cost per request alongside latency.

Standards and prevention

• Introduce design and query review for PRs touching hot paths or the data layer.
• Write standards for migrations, index justification, and what must never run synchronously.
• Instrument data access so issues appear in application traces.
• Contribute CI regression gates and load testing at production-realistic volumes.
• Investigate per-tenant performance in our largest accounts.

What success looks like

Diagnosis: Latency is attributed to real causes with evidence. You tell us things about our systems we didn't know. Investigations into large-account slowness end with an answer, not a theory.

Design: Critical paths have documented designs others can reason about. Store selection is deliberate and written down. Work that doesn't need to be synchronous isn't, and queues behave predictably under load. Feature teams bring you proposals early because you make them better, not slower.

Measurable improvement: Meaningful latency reduction on the highest-cost paths, with before-and-after numbers. No production operation over 1s without a known reason and owner. Data layer utilisation within target at peak, with articulated headroom. Cost per request trending down.

Prevention: Design and query review is standard practice for hot paths. Regressions are caught before merge. Migration standards are followed by teams you don't sit with.

Ultimately: Backend and data layer performance stops being a meaningful source of customer-visible latency, and we can see the next constraint coming instead of discovering it under load.

What we need from you

Experience

• 5+ years building and operating production backend systems, with 3+ where performance and scale were a named part of your job.
• Owned performance for a multi-tenant SaaS product under real load.
• Systems you designed, not only implemented: the alternatives you rejected, the tradeoffs you accepted, and what you'd do differently.
• A specific track record: a problem you diagnosed, its cause, your fix, and what the numbers did.

System design (equal weight with data depth)

What we expect you to reason about:

• Scaling reads — Caching, replicas, indexes, invalidation, staleness
• Scaling writes — Sharding, partitioning, batching, queue buffering
• Read/write separation — Replica routing, CQRS, and when it isn't justified
• Background work — Queues, workers, orchestration, idempotency, dead letters
• Real-time updates — WebSockets, SSE, long polling, connection cost, fan-out
• Traffic spikes — Load balancing, autoscaling, headroom, load shedding
• High availability — Replication, failover, degraded operation
• Reliability — Timeouts, backoff, circuit breakers
• Search — Dedicated indexes instead of the primary store
• Data growth — Partitioning, archival, retention
• Safer deployment — Canary, feature flags, online migrations
• Observability — Logs, metrics, traces and alerts designed in, not added later

We care less about whether you can name these than whether you know when not to use one. Explaining why CQRS was the wrong answer somewhere, and what it cost to find out, tells us more than listing its benefits. You should also be able to write a design doc a team can build from and an executive can follow.

Data systems depth

Real depth in at least two of relational, document, key-value and in-memory stores, with working competence across the rest. Nobody is deep in all four, and we'd rather you say so.

• Relational: execution plans as routine, composite and partial indexes and their write cost, locking, isolation levels, pooling and replica lag.
• Document: embedding vs referencing, shard key selection, aggregation performance and working set.
• Key-value: access-pattern modelling, partition keys, hot partitions, secondary indexes and capacity cost.
• Cache: patterns, eviction and TTL, and system behaviour when the cache is cold or gone.
• Across all: consistency boundaries, and recognising when a query problem is really an application design problem.

Backend engineering

• Production ownership in multiple backend languages, not just familiarity.
• An understanding of how ORMs and ODMs generate queries, and working with them without surprises.
• Concurrency, connection and resource management under load, and queue design, including what happens when a consumer falls behind.

Observability and cloud

• Using APM and tracing tools to find problems, not to confirm assumptions.
• Production experience on a major public cloud, including the cost behaviour of managed data services.
• Instrumenting code so the next person doesn't need you to diagnose the same issue.

Plus points

APM, error tracking and database performance tooling · event streaming or messaging at scale · dedicated search infrastructure · real-time delivery and fan-out at scale · load testing tools and building that capability · noisy-neighbour and tenant isolation · third-party integrations where latency is outside your control · SLOs and error budgets · infrastructure as code · experience as a first dedicated performance hire

What this role is not

• Not a DBA role. Backups, patching and routine operations sit with DevOps. You set performance requirements; you don't run the estate.
• Not greenfield. You'll inherit a production system with schema decisions you didn't make and can't rewrite.
• Not a management track, yet. If the function grows and you want to lead it, that conversation is open.
• Not feature work. Your output is latency removed and regressions prevented.
• Not a ready-made backlog. The team is new, and your first weeks will be spent deciding what matters.

Practicalities

Location: Remote (Pakistan preferred)
Compensation: Market competitive
On-call: You'll join the escalation rota once it's established. Performance and reliability problems tend to be the same problem at different hours.