Pattern: Managing Long-running Tasks

Not everything fits in a request/response. A user clicks “export my data” and generating the file takes 40 seconds. Someone uploads a video that needs transcoding into five resolutions. A campaign sends two million notifications. A request that tries to do this work inline will hold a connection open for far too long, probably hit a gateway timeout, and — worst of all — lose everything if the server crashes halfway. This lesson is the universal pattern for slow work: get it out of the request and onto a queue. It pairs with almost every design that does heavy processing, and it is the natural follow-on to handling large blobs.

In the interview: whenever a step is slow or spiky — media processing, bulk sends, report generation, third-party calls that might hang — say “I’ll do that asynchronously: enqueue a job and return immediately.” Then be ready to talk about how the client finds out when it’s done, and what happens when a worker dies mid-job.

Accept now, process later

The shape of the pattern is three moving parts: an API that enqueues, a durable queue, and a pool of workers that process.

1. The client makes a request. The API validates it, writes a job record (status: queued), pushes a message onto the queue, and immediately returns 202 Accepted with a job_id.
2. A worker from the pool pulls the message, marks the job running, does the work, writes the result (to a database or object storage), and marks it done (or failed).
3. The client discovers the outcome — the subject of the next section.

The queue decouples the fast front end from the slow back end. Traffic spikes are absorbed by the queue depth instead of knocking over the workers, and you scale throughput simply by adding workers (up to the limits of whatever downstream resource they contend on). This is the same buffering idea from scaling writes, applied to units of work rather than individual writes.

How the client learns the result

Two standard mechanisms, often offered together:

• Polling. The client periodically calls GET /jobs/{id} and reads the status until it is done, then fetches the result. Simple and robust; the cost is wasted requests and some latency between completion and the next poll. Use a sensible interval and, ideally, return a suggested retry delay.
• Webhooks / push. When the job finishes, the server calls a URL the client registered, or pushes over an existing WebSocket/SSE connection, or sends a mobile push. This is efficient and near-instant, but the receiver must expose an endpoint and handle duplicate deliveries.

A common production choice is to support both: push for immediacy, with polling as the fallback when a webhook delivery fails.

Making it reliable

A naive queue loses or double-processes work the moment something fails. Real systems lean on a handful of guarantees:

• At-least-once delivery. Messaging systems guarantee a job is delivered at least once, which means it can be delivered more than once — if a worker crashes after doing the work but before acknowledging, the message is redelivered. You accept this and make workers idempotent: processing the same job twice has the same effect as once (use the job_id as a dedup key, or design the write so a repeat is a no-op). Chasing true exactly-once delivery is expensive and usually unnecessary; at-least-once plus idempotency is the pragmatic answer.
• Visibility timeout. When a worker picks up a message, the queue hides it for a timeout instead of deleting it. The worker deletes (acks) it only on success. If the worker dies, the timeout lapses and the message reappears for another worker — so a crash never silently drops the job.
• Retries with backoff. Transient failures (a flaky downstream, a timeout) are retried with exponential backoff and jitter, so a struggling dependency is not hammered.
• Dead-letter queue (DLQ). A “poison” message that keeps failing — malformed input, a permanent bug — must not retry forever and block the queue. After N attempts it is moved to a dead-letter queue for inspection and manual handling, and the pipeline keeps flowing.

worker loop:
  msg = queue.receive()          # becomes invisible for visibility_timeout
  if already_done(msg.job_id):    # idempotency check
      queue.ack(msg); continue
  try:
      do_work(msg)                # the slow operation
      record_result(msg.job_id)
      queue.ack(msg)             # success -> delete the message
  except TransientError:
      pass                       # no ack -> redelivered after backoff
  # after N failures the queue routes msg to the DLQ

Multi-step workflows and orchestration

A single job is easy; a sequence of dependent steps is where it gets interesting — “charge the card, then reserve inventory, then schedule shipping, and if any step fails, undo the previous ones.” Chaining raw queue messages by hand quickly becomes unmanageable: you have to track where each workflow is, retry individual steps, and handle partial failure.

This is what workflow orchestration engines (Temporal, AWS Step Functions, Cadence) are for. They durably track the state of each in-flight workflow, retry individual steps, run steps in parallel where allowed, and — crucially — run compensating actions to undo completed steps when a later one fails. That failure-handling shape is the saga pattern from the distributed-transactions lesson; orchestration is how you run it reliably at scale. For simpler cases you can orchestrate in application code, but once a flow has several steps, timeouts, and compensations, a real workflow engine earns its keep.

Progress and rate control

Two practical touches interviewers like to hear. First, progress tracking: for a long job (encode a 2-hour video, process a 10 GB export), write incremental progress to the job record so the client can show a meaningful progress bar rather than a spinner. Second, rate control: when a job fans out to a rate-limited downstream — sending to an email/SMS provider, calling a partner API — the worker pool must throttle to stay under the limit, often by pacing consumption from the queue rather than draining it as fast as possible.

Where it shows up

This pattern is everywhere a design does heavy or bursty work off the critical path: video transcoding and thumbnailing, sending notifications to millions of users, generating reports and data exports, image processing and ML inference, web crawling, and any batch pipeline. If you catch yourself about to do something slow inside a request handler, that is the signal to reach for a queue and a worker pool.

Key Takeaways

1. Get slow work out of the request. Validate, enqueue, and return 202 with a job id immediately. A queue plus a worker pool decouples a fast front end from slow processing and absorbs spikes in queue depth.

2. Tell the client via polling or webhooks. Polling a status endpoint is simple and robust; webhooks/push are instant but need a dedup-tolerant receiver. Production systems often offer both.

3. Design for at-least-once delivery. Messages can be delivered more than once, so make workers idempotent (dedup on job id). This is far cheaper than chasing true exactly-once.

4. Use visibility timeouts, retries, and a dead-letter queue. A hidden-then-redelivered message survives worker crashes; backoff protects struggling dependencies; the DLQ quarantines poison messages so they do not block the pipeline.

5. Orchestrate multi-step flows. For dependent steps with compensation on failure (the saga shape), use a workflow engine (Temporal, Step Functions) that durably tracks state and retries individual steps, rather than hand-chaining queue messages.