문제
하이브 작업중 다음과 같은 오류가 발생하였습니다. 하이브 테이블의 파티션 정보를 읽어오는 중에 문제가 발생한 것으로 생각되는데 클래스를 보면 kyro.util 관련 클래스를 사용하고 있습니다. 이는 spark에서 사용하는 클래스로 버그로 인해서 문제가 발생하고 있고, 하이브 작업에서 단독으로 작업을 할 때는 필요하지 않습니다.
Serialization trace:
partSpec (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:463)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:302)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:268)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:484)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:477)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:715)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
at java.security.AccessController.doPrivileged(AccessController.java:770)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:169)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: column_name
Serialization trace:
partSpec (org.apache.hadoop.hive.ql.plan.PartitionDesc)
aliasToPartnInfo (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:599)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:134)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:648)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.read(FieldSerializer.java:605)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:221)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:626)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1082)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:973)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:987)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:423)
... 13 more
Caused by: java.lang.ClassNotFoundException: column_name
at java.lang.Class.forNameImpl(Native Method)
at java.lang.Class.forName(Class.java:407)
at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
... 28 more해결 방법
mapred-site.xml파일의 mapreduce.application.classpath에서 spark 관련 라이브러리 위치를 삭제 합니다.
반응형