AWS Kinesis python으로 다뤄보기 Working with AWS Kinesis using Python

Python-kinesis

The process of connecting to Kinesis Datastream in Python to send/receive data

When sending data to Kinesis, you must send it as bytes or bytearray (you need to encode with encode(‘utf-8’)).

1. Requirements

   • Install boto3, kinesis

       pip3 install -U pip
       python3 -m pip install boto3 kinesis
     

2. Setting Credential and config

   Create the two files below.

   Place the credentials and config files in ~/.aws

   ~/.aws/

   • credentials –> setting access_key, secret_key

       [default]
       aws_access_key_id=
       aws_secret_access_key=
     

   • config –> setting region, type

       [default]
       region=region_info
       output=json
     

3. Test setting

   • test

       import boto3
       from kinesis.producer import KinesisProducer
       from kinesis.consumer import KinesisConsumer
     
       consumer = KinesisConsumer(stream_name=env.KINESIS_STREAM_NAME)
       for message in consumer:
           print("############")
           print(message)
           print("############")
     

   • test consumer

       from kinesis.consumer import KinesisConsumer
       from kinesis.state import DynamoDB
     
       consumer = KinesisConsumer(stream_name='my-stream', state=DynamoDB(table_name='my-kinesis-state'))
       for message in consumer:
           print ("Received message: {0}".format(message))
     

   • test producer

       from kinesis.producer import KinesisProducer
     
       producer = KinesisProducer(stream_name='my-stream')
       producer.put('Hello World from Python')
     

---

REFERENCE

• python-kinesis document (Link)
• python-kinesis document github (Link)
• aws boto3/kinesis configure documentation (Link)
• python-kinesis & python-s3 example Blog(Link)
• aws boto3 data stream document(Link)
• boto3 - how to credential –> credential process (Link)