AWS Kinesis python으로 다뤄보기

Python-kinesis

Python에서 Kinesis Datastream에 연동하여 데이터를 주고/받는 과정

Kinesis로 데이터를 전달할 때, byte혹은 bytearray로 전달 해야 함 (encode(‘utf-8’)로 전달 해 주어야 함)

1. Requirements

   • Install boto3, kinesis

       pip3 install -U pip
       python3 -m pip install boto3 kinesis
     

2. Setting Credential and config

   아래 두개의 파일을 생성한다.

   credentials, config 두개의 파일을 ~/.aws에 위치시킨다

   ~/.aws/

   • credentials —> setting access_key, sercret_key

       [default]
       aws_access_key_id=
       aws_secret_access_key=
     

   • config —> setting region, type

       [default]
       region=region_info
       output=json
     

3. Test setting

   • test

       import boto3
       from kinesis.producer import KinesisProducer
       from kinesis.consumer import KinesisConsumer
     
       consumer = KinesisConsumer(stream_name=env.KINESIS_STREAM_NAME)
       for message in consumer:
           print("############")
           print(message)
           print("############")
     

   • test consumer

       from kinesis.consumer import KinesisConsumer
       from kinesis.state import DynamoDB
     
       consumer = KinesisConsumer(stream_name='my-stream', state=DynamoDB(table_name='my-kinesis-state'))
       for message in consumer:
           print ("Received message: {0}".format(message))
     

   • test producer

       from kinesis.producer import KinesisProducer
     
       producer = KinesisProducer(stream_name='my-stream')
       producer.put('Hello World from Python')
     

---

REFERENCE

• python-kinesis document (Link)
• python-kinesis document github (Link)
• aws boto3/kinesis configure documentation (Link)
• python-kinesis & python-s3 example Blod(Link)
• aws boto3 data stream document(Link)
• boto3 - how to credential —> credential process (Link)