울이

Love Code. Work. Sports.

4 분 소요

이번 글의 내용은 Apache Flink 책인 <Stream Processing with Apache Flink: Fundamentals, Implementation, and Operation of Streaming Applications> 를 참고했으며 직접 개발환경을 세팅 해 보고 작성했다.

Docker를 기반으로 작업을 하려고 한다. Docker를 다룰 줄 알거나 설치가 되어 있다는 가정 하에 작업을 해 보려고 한다

플링크는 스파크 다음 세대의 빅데이터 분석 프레임워크로서 짧은 지연시간 내에 스트림 데이터를 처리하면서도 강력한 상태 관리가 필요한 경우에 최적의 선택이 될 수 있다.

이번에는 설치나 세팅이 비교적 간편한 Docker를 사용해서 Apache Flink를 실행을 해 보았다.

  • Master Process

    → Job Manager

    → Flink의 마스터 노드. Flink내의 Task들을 실행하는 Worker들을 관리하는 노드.

    → Task를 스케쥴링하여 실행 시점을 관리하며 실행 상황을 모니터링.

    → 문제가 생겼을 경우 Recover를 담당.

  • Worker Process

    → Task Manager.

    → Master Process에게 할당 받은 Task들을 처리.

    → 작성 된 Job App (JAR파일) 실행을 담당.

Docker에서 Apache Flink를 띄웠다. 위에서 설명 된 것 처럼 1개의 Master Process를 띄우고, Worker Process는 입맛에 맞게, 내가 원하는 그림대로 띄워보도록 하자. (올릴 job이 많으면 worker를 늘리면 될 것 같다.)

나는 Mac M1을 사용한다. 때문에 Intel과 M1 모두 다 명령어를 정리 해 보았다. (m1은 아직 찾기가 힘들더라..) Window 라면 Intel 따라서 하시면 될 것 같고, Linux는 amd64를 사용하시면 될 것 같다. (이미지 태그는 조금 차이가 있을수도 있다.)

Intel Mac

  • Master Process (Job Manager)

      docker run -d --name flink-jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 flink jobmanager

  • Worker Process (Task Manager)

      # task manager 1
  docker run -d --name flink-taskmanager-1 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager flink taskmanager

  # task manager 2
  docker run -d --name flink-taskmanager-2 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager flink taskmanager

M1 Mac

  • Master Process (Job Manager)

      docker run -d --name flink-jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 arm64v8/flink jobmanager

  • Worker Process (Task Manager)

      # task manager 1
  docker run -d --name flink-taskmanager-1 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager arm64v8/flink taskmanager

  # task manager 2
  docker run -d --name flink-taskmanager-2 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager arm64v8/flink taskmanager

Linux

  • Master Process (Job Manager)

      docker run -d --name flink-jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 amd64/flink jobmanager

  • Worker Process (Task Manager)

      # task manager 1
  docker run -d --name flink-taskmanager-1 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager amd64/flink taskmanager

  # task manager 2
  docker run -d --name flink-taskmanager-2 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager amd64/flink taskmanager

결과화면

나는 task manager를 3개까지 올렸다. (중간에 있는 flink-taskmanager를 1~3 까지 생성)

나는 m1 Mac을 사용하는데, docker의 Flink 중 m1 mac과 호환되는 Flink는 버전이 좀 낮은 것 같았다. Docker가 아닌 Flink 공식 홈페이지에서 파일을 받으면, Flink-1.13.0 버전이라서 화면 구성이 조금 다르다.

The content of this post references the Apache Flink book <Stream Processing with Apache Flink: Fundamentals, Implementation, and Operation of Streaming Applications> and was written after setting up the development environment myself.

I’m going to work based on Docker. I’ll proceed assuming you know how to use Docker or have it installed.

Flink is the next-generation big data analysis framework after Spark and can be the optimal choice when you need to process stream data with low latency while requiring powerful state management.

This time, I ran Apache Flink using Docker, which is relatively easy to install and set up.

  • Master Process

    -> Job Manager

    -> Flink’s master node. A node that manages Workers executing Tasks within Flink.

    -> Manages execution timing by scheduling Tasks and monitors execution status.

    -> Responsible for Recovery when problems occur.

  • Worker Process

    -> Task Manager.

    -> Processes Tasks assigned by the Master Process.

    -> Responsible for executing the written Job App (JAR file).

I ran Apache Flink on Docker. As explained above, let’s run 1 Master Process, and run Worker Processes according to your preferences and desired configuration. (If you have many jobs to run, you can increase the workers.)

I use Mac M1. So I’ve organized commands for both Intel and M1. (M1 was still hard to find..) If you’re on Windows, you can follow Intel, and for Linux, you can use amd64. (Image tags may differ slightly.)

Intel Mac

  • Master Process (Job Manager)

      docker run -d --name flink-jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 flink jobmanager

  • Worker Process (Task Manager)

      # task manager 1
  docker run -d --name flink-taskmanager-1 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager flink taskmanager

  # task manager 2
  docker run -d --name flink-taskmanager-2 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager flink taskmanager

M1 Mac

  • Master Process (Job Manager)

      docker run -d --name flink-jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 arm64v8/flink jobmanager

  • Worker Process (Task Manager)

      # task manager 1
  docker run -d --name flink-taskmanager-1 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager arm64v8/flink taskmanager

  # task manager 2
  docker run -d --name flink-taskmanager-2 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager arm64v8/flink taskmanager

Linux

  • Master Process (Job Manager)

      docker run -d --name flink-jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 amd64/flink jobmanager

  • Worker Process (Task Manager)

      # task manager 1
  docker run -d --name flink-taskmanager-1 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager amd64/flink taskmanager

  # task manager 2
  docker run -d --name flink-taskmanager-2 --link flink-jobmanager:jobmanager -e JOB_MANAGER_RPC_ADDRESS=jobmanager amd64/flink taskmanager

Result Screen

I ran up to 3 task managers. (Created flink-taskmanager 1 through 3)

I use M1 Mac, and the Flink version compatible with M1 Mac in Docker seemed to be a bit older. If you download the file from the official Flink website instead of Docker, it’s Flink-1.13.0 version, so the screen layout is slightly different.

댓글남기기

참고