Apache Kafka ASCII Arts

This blog post shows some ASCII arts describing the concepts of Apache Kafka.

I made them just for fun :) Enjoy!

Consume Group

+-----------------------+     +------------------------------------------------------------------------------------+
|                       |     |                                                                                    |
| +-------------------+ |     |                                                                append-only         |
| |                   | |     |                  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                                   |
| | consumer C1       | |     |     partition P1 | | | | | | | | | | | | | | | |    <---------------+              |
| |                   | |     |                  +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                    |              |
| +-------------------+ |     |                                                                     |              |
|                       |     |                  +-+-+-+-+-+-+-+-+-+-+-+-+                          |              |
|                       |     |     partition P2 | | | | | | | | | | | | |  <------------------------------------------
| +-------------------+ |     |                  +-+-+-+-+-+-+-+-+-+-+-+-+                          |              |
| |                   | |     |                                                                     |              |  multiple producers
| | consumer C2       | |     |                  +-+-+-+-+-+-+-+-+-+-+-+                            |              |  write messages
| |                   | |     |     partition P3 | | | | | | | | | | | |    <-----------------------+              |
| +--------------+----+ |     |                  +-+-+-+-+-+-+---+-+-+-+                                           |
|                |      |     |                                ^  offset(C1, P3)                                   |
|                |      |     |                                |                                                   |
+-----------------------+     +------------------------------------------------------------------------------------+
                 |                                             |
                 |                                             |
                 |                                             |
                 +---------------------------------------------+

                                     consumers poll messages from partitions

Partition <-> Consumer

+--------------+                         +--------------+
|              |                         |              |
| +----------+ |                         | +----------+ |
| |    P1    +--------------------+-------->    C1    | |
| +----------+ |                  |      | +----------+ |
|              |                  |      |              |
| +----------+ |                  |      | +----------+ |
| |    P2    +-----------------+----------->    C2    | |
| +----------+ |               |  |      | +----------+ |
|              |               |  |      |              |
| +----------+ |               |  |      | +----------+ |
| |    P3    +-----------+----------------->    C3    | |
| +----------+ |         |     |  |      | +----------+ |
|              |         |     |  |      |              |
| +----------+ |         |     |  |      | +----------+ |
| |    P4    +--------+-------------------->    C4    | |
| +----------+ |      |  |     |  |      | +----------+ |
|              |      |  |     |  |      |              |
+--------------+      |  |     |  |      +--------------+
                      |  |     |  |
                      |  |     |  |      Consumer Group CG2
                      |  |     |  |      +--------------+
                      |  |     |  |      |              |
                      |  |     |  |      | +----------+ |
                      |  |     +--+-------->    C1    | |
                      |  |               | +----------+ |
                      |  |               |              |
                      |  |               | +----------+ |
                      +--+----------------->    C2    | |
                                         | +----------+ |
                                         |              |
                                         +--------------+

Producer

              +---------------+
              |     topic     |
              +---------------+
              |   partition   |
              +---------------+
              |      key      |
              +---------------+
              |     value     |
              +-------+-------+
                      |
                      | Send()
                      |
                      v                                      e.g. Apache Avro
              +-------+-------+      if any               +------------------+
              |               +-------------------------->+                  |
              |  Serializer   |                           | schema registry  |
              |               +<--------------------------+                  |
              +-------+-------+      cache locally        +------------------+
                      |
                      |
                      |
                      v
              +-------+-------+
              |               | based on key
              |  Partitioner  | (consistent hashing)
              |               |
              +-------+-------+
                      |
         +------------+-------------+
Topic T1 |                          | Topic T2
+--------v---------+      +---------v--------+
|                  |      |                  |
|  Partition P1    |      |  Partition P2    |
|                  |      |                  |
|  +------------+  |      |  +------------+  |
|  | Batch B1   |  |      |  | Batch B2   |  |
|  +------------+  |      |  +------------+  |
|                  |      |                  |
+--------+---------+      +---------+--------+
         |                          |
         |                          |
         |    +---------------+     |
         |    |               |     |
         +--->+   Broker      +<----+
              |               |
              +---------------+