This page is part of the ForgeSDLC knowledge base — an AI-assisted, human-directed methodology for taking product work from concept to production. For the core operating model and vocabulary, see Forge SDLC overview and What is ForgeSDLC?.

Lambda, Kappa & unified data architectures

Purpose: Project-agnostic guide to batch–stream hybrid and unified streaming architectures: how they evolved, how they differ, and how to choose among them — including lakehouse as a common unified-storage pattern.

Audience: Teams using blueprints/disciplines/data/bigdata/. Pair with BIGDATA.md §1 (principles) for the broader decision framework.

1. Overview: from batch-only to unified streaming

Early big-data systems were batch-only: periodic jobs recomputed views from full or incremental dumps. That model is simple and can be very accurate for historical analytics, but latency is bounded by the batch interval.

Lambda architecture added a speed layer so low-latency views could coexist with a batch layer that recomputes authoritative state. Kappa architecture questioned whether two paths were necessary: if the immutable event log is the system of record, a single stream-processing stack can often replace batch by replaying the log.

Unified approaches (modern stream processors with batch APIs, or lakehouse table formats with ACID + time travel) aim to one engine and one storage abstraction for both reprocessing and serving, reducing operational duplication while preserving flexibility.

2. Lambda architecture (deep dive)

Lambda splits work across three conceptual layers:

When it shines: You need provably correct historical recomputation (e.g., late-arriving facts, complex corrections) and sub-batch latency for some use cases.

3. Kappa architecture (deep dive)

Kappa removes the separate batch path: stream processing consumes an append-only log; reprocessing means resetting offsets or replaying topics through the same topology (often with upgraded code).

When it shines: The domain can be modeled as events, idempotent sinks exist, and replay is an acceptable substitute for heavyweight batch recompute.

4. Comparison matrix: Lambda vs Kappa vs unified

5. Decision flowchart (architecture choice)

6. Technology mapping (illustrative)

Tools evolve: treat this table as family resemblance, not a rigid taxonomy.

7. Lakehouse: lake + warehouse concerns

A lakehouse keeps cheap object storage as the main store while adding warehouse-like features: transactions, schema enforcement, time travel, and incremental processing against open table formats.

Lakehouse does not by itself replace organizational patterns (e.g., data mesh); it is primarily a storage and execution unification play.

8. Data quality by architecture

9. Migration patterns

10. Anti-patterns

11. External references

Keep project-specific data architecture decisions in docs/adr/ and pipeline documentation in docs/development/, not in this file.