[Tez] Tez 0.10.3과 Hadoop 3.3.6을 연결할 때 발생하는 오류

Tez 0.10.3 와 Hadoop 3.3.6을 설치하니 다음과 같은 오류가 발생하였습니다.

 

2024-09-05T06:53:52,466  INFO [main] conf.HiveConf: Using the default value passed in for log id: 693153d4-f980-4b4a-84b9-f55371e343d9
2024-09-05T06:53:52,471  INFO [main] session.SessionState: Error closing tez session
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:218) ~[hive-exec-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.toString(TezSessionState.java:166) ~[hive-exec-3.1.3.jar:3.1.3]
    at java.lang.String.valueOf(String.java:2994) ~[?:1.8.0_262]
    at java.lang.StringBuilder.append(StringBuilder.java:131) ~[?:1.8.0_262]
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.closeIfNotDefault(TezSessionPoolManager.java:351) ~[hive-exec-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1765) [hive-exec-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66) [hive-cli-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:762) [hive-cli-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) [hive-cli-3.1.3.jar:3.1.3]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_262]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_262]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_262]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]
    at org.apache.hadoop.util.RunJar.run(RunJar.java:328) [hadoop-common-3.3.6.jar:?]
    at org.apache.hadoop.util.RunJar.main(RunJar.java:241) [hadoop-common-3.3.6.jar:?]
Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto
    at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_262]
    at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_262]
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:209) ~[hive-exec-3.1.3.jar:3.1.3]
    ... 14 more
Caused by: java.lang.Exception: java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:381) ~[hive-exec-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:371) ~[hive-exec-3.1.3.jar:3.1.3]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262]
Caused by: java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto
    at org.apache.tez.dag.api.records.DAGProtos$ConfigurationProto.getSerializedSize(DAGProtos.java:21636) ~[tez-api-0.10.3.jar:0.10.3]
    at com.google.protobuf.AbstractMessageLite.writeTo(AbstractMessageLite.java:75) ~[hive-exec-3.1.3.jar:3.1.3]
    at org.apache.tez.client.TezClientUtils.createApplicationSubmissionContext(TezClientUtils.java:571) ~[tez-api-0.10.3.jar:0.10.3]
    at org.apache.tez.client.TezClient.setupApplicationContext(TezClient.java:500) ~[tez-api-0.10.3.jar:0.10.3]
    at org.apache.tez.client.TezClient.start(TezClient.java:400) ~[tez-api-0.10.3.jar:0.10.3]
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:433) ~[hive-exec-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.access$100(TezSessionState.java:101) ~[hive-exec-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:376) ~[hive-exec-3.1.3.jar:3.1.3]
    at org.apache.hadoop.hive.ql.exec.tez.TezSessionState$1.call(TezSessionState.java:371) ~[hive-exec-3.1.3.jar:3.1.3]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_262]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_262]

 

해당 이슈는 현재 지라에 등록되어 있는 이슈로 Hadoop 3.3.5 버전부터 protobuf 버전이 변경되어 발생하고 있습니다.

 

https://issues.apache.org/jira/browse/HADOOP-19174

[HADOOP-19174] Tez and hive jobs fail due to google's protobuf 2.5.0 in classpath - ASF JIRA

 

현재까지는 해결방법이 없어서 Tez 0.10.2 로 변경하여 테스트를 진행하였습니다.