Concurrent Processing System

Scenario

Build a system that processes a stream of work items (image resizing, text analysis, or any CPU-meaningful task of your choice) across multiple workers, with bounded concurrency, structured cancellation, and enough observability to diagnose a problem after the fact rather than only while watching it live.

Requirements

Core

• Work is distributed across multiple workers (worker_threads in Node, or Web Workers in the browser), not processed serially on the main thread.
• A bounded queue (recall backpressure) limits how much pending work accumulates — the system must not silently grow an unbounded queue under sustained heavy load.
• Cancelling the overall operation propagates to every in-flight and queued item — structured concurrency (recall Structured Concurrency), not independently-running orphaned work.
• Structured logging with a correlation id per work item, sufficient to reconstruct that item’s full processing path (queued → assigned to a worker → completed or failed) after the fact from logs alone.
• At least one metric tracked over the run (throughput, queue depth over time, or error rate) and reported at the end.

Load and failure testing

• A load test demonstrating behavior under sustained heavy input — specifically confirming the bounded queue actually bounds (memory/queue depth stabilizes, doesn’t grow unboundedly).
• Deliberate failure injection: some fraction of work items simulate a failure (a thrown error, a timeout), and the system’s handling of that (retry per Retry with Exponential Backoff, dead-letter, or documented drop) is demonstrated, not just theoretically described.

Stretch

• A live dashboard or periodic console summary showing queue depth, throughput, and error rate during a run.
• Graceful shutdown: an in-progress run can be stopped cleanly, with in-flight work either completed or cleanly cancelled, never left in an ambiguous state.

Reflect

• What did the failure-injection testing reveal that wasn’t obvious from reading the code? Did anything surprise you about how cancellation or backpressure actually behaved under real load?
• If this system needed to scale to ten times the throughput, what part of the current design would be the first bottleneck?

Suggested milestones

1. A working single-worker pipeline, no bounding or cancellation yet — get the core processing correct first.
2. Add multiple workers and a bounded queue; verify the bound actually holds under load.
3. Add structured cancellation, propagating to in-flight and queued work.
4. Add structured logging with correlation ids and at least one tracked metric.
5. Run the load test and the failure-injection test; record and document the evidence.

Acceptance checks

• Under sustained load well beyond normal capacity, queue depth (or memory usage) stabilizes rather than growing without bound.
• Cancelling a run partway through leaves no orphaned worker still processing minutes later.
• Given only the structured logs from a run, a specific work item’s full processing history can be reconstructed without needing to have watched the run live.
• The failure-injection test demonstrates the system’s actual documented failure-handling behavior, not just its happy path.