[flink] flink를 yarn session 모드로 동작할 때 작업이 ACCEPTED 상태로 대기 할 때 해결 방법

flink를 YARN에서 동작할 때 작업이 ACCEPTED 상태로 대기하면서 다음과 같은 로그가 출력되는 경우가 있습니다.

2022-04-21 16:40:32,601 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-04-21 16:40:32,853 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-04-21 16:40:33,104 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
2022-04-21 16:40:33,356 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

 

yarn 의 로그를 확인해 보니 다음과 같은 로그가 출력되고 있었습니다. 8030 포트는 리소스매니저 스케줄러 주소가 설정되지 않아서 오류가 발생하고 있습니다.

2022-04-21 15:47:48,785 INFO  org.apache.hadoop.ipc.Client                                 [] - Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2022-04-21 15:47:49,785 INFO  org.apache.hadoop.ipc.Client                                 [] - Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2022-04-21 15:47:50,786 INFO  org.apache.hadoop.ipc.Client                                 [] - Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
2022-04-21 15:47:51,787 INFO  org.apache.hadoop.ipc.Client                                 [] - Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)

해결 방법

flink-conf.yaml 에 리소스 매니저 설정을 추가하면 됩니다.

flink.hadoop.yarn.resourcemanager.scheduler.address: test-server.com:8030