[hive] rank() 함수 처리중 Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException

hive 쿼리중 rank() 함수를 이용하였는데 다음과 같은 오류가 발생하였다. 

json을 UDTF를 이용하여 변환하는 함수 였는데 오류가 발생하였다. 

그래서 cbo를 false 로 설정했더니 처리 되었다. 

explain 으로 확인결과 Reduce 단계가 하나 빠진것이 해결된 것 같다. 

정확한 원인은 알 수 없으나 검색결과는 bug 일 가능성도 있는 것 같다. 

혹시 모르니 cbo 를 false 로 설정해보길 권한다. 

set hive.cbo.enable=false;

Caused by: java.lang.RuntimeException: Failed to load plan:

org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 6104169, Size: 254

Serialization trace:

_mainHash (org.codehaus.jackson.sym.BytesToNameCanonicalizer)

_rootByteSymbols (org.codehaus.jackson.JsonFactory)

jsonFactory (brickhouse.udf.json.ToJsonUDF)

genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)

chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)

colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)

childOperators (org.apache.hadoop.hive.ql.exec.GroupByOperator)

reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)

at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:451)

at org.apache.hadoop.hive.ql.exec.Utilities.getReduceWork(Utilities.java:306)

at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor$1.call(ReduceRecordProcessor.java:106)

at org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:75)

... 16 more

Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 6104169, Size: 254

Serialization trace:

_mainHash (org.codehaus.jackson.sym.BytesToNameCanonicalizer)

_rootByteSymbols (org.codehaus.jackson.JsonFactory)

jsonFactory (brickhouse.udf.json.ToJsonUDF)

genericUDF (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)

chidren (org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc)

colExprMap (org.apache.hadoop.hive.ql.exec.SelectOperator)

childOperators (org.apache.hadoop.hive.ql.exec.GroupByOperator)

reducer (org.apache.hadoop.hive.ql.plan.ReduceWork)

at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144)

at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)

at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:759)

at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObjectOrNull(SerializationUtilities.java:198)

at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:132)

at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)