hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yehuda Finkelstein <yeh...@veracity-group.com>
Subject Error: java.lang.IllegalArgumentE:Column has wrong number of index entries found - when trying to insert from JSON external table to ORC table
Date Tue, 08 Sep 2015 12:28:38 GMT
Hi all,



We have a process that load data from folder that have json files.

We use external table on top of the json folder and then insert into ORC
table.



We get the error below during the load

“Column has wrong number of index entries found: 0 expected: 20”





I validate the JSON data and it look ok.

I notice that I got this error only when I reach amount of file (>40).

I was able to complete the insert when the number of files is less than 40.

I was able to load the data after concatenate all files into one big file.



We are using this method for almost a year and never get this error before,

even while we load folder with thousands of files.





Task ID:

  task_1439882549481_118182_m_000000



URL:


http://hdname:8088/taskdetails.jsp?jobid=job_1439882549481_118182&tipid=task_1439882549481_118182_m_000000

-----

Diagnostic Messages for this Task:

Error: java.lang.IllegalArgumentException: Column has wrong number of index
entries found: 0 expected: 20

                at
org.apache.hadoop.hive.ql.io.orc.WriterImpl$TreeWriter.writeStripe(WriterImpl.java:726)

                at
org.apache.hadoop.hive.ql.io.orc.WriterImpl$StructTreeWriter.writeStripe(WriterImpl.java:1614)

                at
org.apache.hadoop.hive.ql.io.orc.WriterImpl.flushStripe(WriterImpl.java:1996)

                at
org.apache.hadoop.hive.ql.io.orc.WriterImpl.close(WriterImpl.java:2288)

                at
org.apache.hadoop.hive.ql.exec.OrcFileMergeOperator.closeOp(OrcFileMergeOperator.java:215)

                at
org.apache.hadoop.hive.ql.io.merge.MergeFileMapper.close(MergeFileMapper.java:98)

                at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)

                at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)

                at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

                at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

                at java.security.AccessController.doPrivileged(Native
Method)

                at javax.security.auth.Subject.doAs(Subject.java:415)

                at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

                at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)





FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.io.merge.MergeFileTask

MapReduce Jobs Launched:

Stage-Stage-1: Map: 2   Cumulative CPU: 55.43 sec   HDFS Read: 266399137
HDFS Write: 17963167 SUCCESS

Stage-Stage-2: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL

Total MapReduce CPU Time Spent: 55 seconds 430 msec









Regard,

Yehuda





*Yehuda Finkelshtein*

*Senior DBA consultant*,

Veracity Ltd.

yehuda@veracity-group.com

www.veracity-group.com

[image: Description: Description: cid:image001.gif@01C8A535.5F5A43E0]

Mime
View raw message