An event streaming platform is a system that moves data as it happens, in the order it happens, and stores it long enough that anyone who cares can read it — including readers who arrived an hour, a day, or a week later. The simplest mental model is this: it is a giant, append-only log file that many producers can write to and many consumers can read from independently. Every event has a timestamp and a position; nothing is ever overwritten; and consumers track their own place in the log.

That sounds mundane, but it is the architectural foundation for almost everything modern in data infrastructure. When you order from Uber and the driver's location updates on your screen, that is event streaming. When a fraud-detection system blocks your credit card mid-transaction, that is event streaming. When your company syncs Salesforce changes into a data warehouse within seconds, that is event streaming. The pattern is the same: something happened, an event was emitted, a log captured it, and one or more downstream systems reacted.

The Most Important Distinction: Open-Source Project vs Commercial Vendor

Before going further, here is the distinction that this wiki insists on and most other resources blur: the open-source project and the commercial company that sells it are not the same thing. Conflating them leads to muddled architectural decisions and worse vendor evaluations.

In streaming, the most important version of this distinction is:

• Apache Kafka is the open-source project, governed by the Apache Software Foundation, licensed Apache 2.0, and downloadable for free.
• Confluent is the company founded in 2014 by Kafka's original creators to commercialize Kafka. It sells a managed cloud service (Confluent Cloud), an enterprise distribution (Confluent Platform), and a constellation of value-add tooling (Schema Registry, Connect, Flink integration).
• AWS MSK is Amazon's managed version of the same Apache Kafka project — different vendor, same underlying technology. AWS Kinesis is something else entirely (more on this in a moment).
• Redpanda is a Kafka-compatible rewrite — it speaks the same wire protocol as Kafka but is built from scratch in C++ with a completely different implementation. From a client's perspective it looks like Kafka; under the hood it is not.

The same pattern holds for Pulsar:

• Apache Pulsar is the open-source project, originally created at Yahoo and now governed by the ASF.
• StreamNative is the company founded by Pulsar's original creators to commercialize it — the Confluent of Pulsar, with all the structural challenges that implies.

If you are trying to understand who is winning, who is at risk, and what you should actually deploy, you have to think about the project and the vendor separately. The project's adoption is one question. The commercial viability of any single vendor selling that project is a different question.

OSS Projects to Commercial Vendors — The Streaming Map

OSS project / standard	Commercial vendor(s)	Notes
—-	—-	—-
Apache Kafka	Confluent, AWS MSK, Aiven, Instaclustr	The dominant standard. Multiple vendors all selling managed versions of the same Apache project.
Apache Kafka (wire protocol only)	Redpanda, WarpStream (now Confluent)	Kafka-compatible rewrites. Speak the protocol, reimplement the broker.
Apache Pulsar	StreamNative, DataStax (Astra Streaming)	Two commercial vendors splitting a smaller pie.
AWS Kinesis (proprietary)	AWS only	Amazon's homegrown service. Not Kafka-based.
Google Pub/Sub (proprietary)	Google only	Google's homegrown service. Not Kafka-based.
Azure Event Hubs	Microsoft only	Speaks a Kafka-compatible API but is Microsoft's proprietary backend.

A Note on AWS Kinesis: It Is Not Kafka

This is a common point of confusion that this wiki will spell out plainly: AWS Kinesis is not Kafka, and it is not Kafka-compatible. It is Amazon's own proprietary streaming service, with its own API, its own client libraries, and its own conceptual vocabulary.

Kinesis terminology even differs from Kafka in ways that catch people off guard:

• Kafka has topics; Kinesis has streams.
• Kafka has partitions; Kinesis has shards.
• Kafka has consumer groups; Kinesis has enhanced fan-out consumers with a different model.
• Kafka offsets are 64-bit integers per partition; Kinesis has sequence numbers with different semantics.

If you switch from Kafka to Kinesis (or the reverse), you are rewriting your producer and consumer code, not just changing connection strings. The two are conceptually similar — both are durable, partitioned, replayable logs — but they are not interchangeable.

The other AWS streaming product, AWS MSK (Managed Streaming for Apache Kafka), is genuine Kafka. MSK runs the actual Apache Kafka project on AWS-managed infrastructure. If you want Kafka on AWS without managing brokers yourself, MSK is the AWS answer; Kinesis is a different product for a different purpose.

Streams vs Queues: Why Kafka Beat RabbitMQ

The most important conceptual distinction in this category is streams vs. queues, and most people get it wrong.

A traditional queue (RabbitMQ, ActiveMQ, Amazon SQS) is like a to-do list. A producer writes a message; a consumer reads it; the message is deleted. If two consumers want the same message, you have to set up a fan-out exchange. If a consumer crashes after reading but before processing, you need acknowledgments and dead-letter queues. The queue is a transport, not a storage system — once the message is delivered, it is gone.

A stream (Kafka, Kinesis, Pulsar) is like a tape recording. The producer appends events to the tape; the tape never erases. Any number of consumers can play the tape from any starting point, at their own speed, independently of each other. If a new analytics team shows up next year and wants the last 30 days of data, they just rewind. The stream is both the transport and the durable record.

The streaming model won because modern data systems are fundamentally about decoupling producers from consumers across time. In 2010, if your e-commerce backend wanted to feed five downstream systems (warehouse, search index, recommendation engine, fraud detector, email service), you wrote five custom integrations and prayed nothing broke. After Kafka, you publish one event ("OrderPlaced") to one topic, and each consumer reads it on their own schedule. The integration count dropped from N times M to N plus M.

Key Concepts, Explained Simply

Topics, partitions, and offsets. A topic is a named stream (like user-clicks or order-events). A topic is split into partitions, which are the unit of parallelism — each partition is a separate ordered log, and consumers can read partitions in parallel. Within a partition, every event has an offset: a monotonically increasing number that identifies its position. Consumers track which offsets they have processed.

Producers and consumers. A producer writes events to topics. A consumer reads them. Consumers are organized into consumer groups — if five consumer instances are in the same group, Kafka balances partitions across them so each event is processed exactly once by the group. Different groups (e.g., "warehouse-loader" vs. "fraud-detector") read the same events independently.

Retention. Unlike a queue, a stream keeps events for a configurable retention period — hours, days, or forever. This is the magic that makes streams replayable. A bug in your downstream service? Reset the offset and reprocess. A new analytics use case? Read from the beginning of the topic.

Schema registry. Producers and consumers need to agree on the shape of events. A schema registry stores Avro/Protobuf/JSON Schema definitions for each topic, enforces compatibility rules, and prevents producers from breaking consumers with backward-incompatible changes. Confluent's was the first widely used one and remains the de facto standard.

Why "log" is the magic word. In a database, the write-ahead log is an internal implementation detail — the canonical thing is the table. Jay Kreps's insight in the early 2010s was: what if we make the log the canonical thing, and treat tables as derived views of the log? That single inversion is the foundation of Kafka, change data capture, event sourcing, and the entire "streaming-first" architecture. If you read one essay on this category, read Kreps's "The Log: What every software engineer should know about real-time data's unifying abstraction."