Kafka로 메시지와 이벤트 처리하기 - (2) Python으로 consumer, producer 만들기 Message and Event Processing with Kafka (2) - Python Consumer and Producer

This post was written while reading the book Kafka, The Definitive Guide. For simple execution and example programming, I referred to the official documentation rather than the book.

Apache Kafka

Publishing Kafka Topics and Creating Producer and Consumer Programs - Python

Let’s create a new Topic. We’ll create a Topic called blog-post to use as an example.

bin/kafka-topics.sh --create --topic blog-post --bootstrap-server localhost:9092

Setting Up the Development Environment

• Setting up a virtual environment

virtualenv venv
source venv/bin/activate

• Installing the kafka module

(venv) $ pip3 install kafka-python

producer.py

I used the time module to measure how long it takes to send 10,000 messages to the Queue.

import time
from kafka import KafkaProducer

# Create producer object
# acks 0 -> prioritize fast transmission, acks 1 -> prioritize data accuracy
producer = KafkaProducer(acks=0, compression_type='gzip',bootstrap_servers=['localhost:9092'])

start = time.time()

for i in range(10000):
 producer.send('blog-post',b'Kafka Blog Post Event Message')
 producer.flush() # Send data in the queue

end = time.time() - start
print(end)

consumer.py

from kafka import KafkaConsumer, consumer

# Create consumer object
consumer = KafkaConsumer(
    'blog-post',
    bootstrap_servers=['127.0.0.1:9092'],
    auto_offset_reset='earliest',
    enable_auto_commit=True,
    consumer_timeout_ms=1000
)

while True:
    for message in consumer:
        print(message.topic, message.partition, message.offset, message.key, message.value)

Execution

• Since we created the Topic before starting, we just need to run producer.py and consumer.py.
• producer.py generates data and sends 10,000 messages to the ‘blog-post’ Topic.
• consumer.py reads data, and since we created an infinite loop, it prints messages as they are received.

> python producer.py
3.8476390838623047

> python consumer.py
...
blog-post 0 100 None b'Kafka Blog Post Event Message'
blog-post 0 101 None b'Kafka Blog Post Event Message'
blog-post 0 102 None b'Kafka Blog Post Event Message'
blog-post 0 103 None b'Kafka Blog Post Event Message'
...