hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prasanth Jayachandran <pjayachand...@hortonworks.com>
Subject Re: RES: Join error with ORC Hive tables
Date Thu, 02 Oct 2014 19:53:21 GMT
Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001.
Invalid postscript.

>From the exception it looks like the file it is trying to read from table stored as ORC
but the above file is not an ORC file.  How is the ORC table loaded? My suspicion is some
non-ORC file ended up in ORC table.

Thanks
Prasanth Jayachandran

On Oct 2, 2014, at 12:26 PM, Thiago Henrique dos Santos Bento <thiago.bento@b2wdigital.com>
wrote:

> Erro:
> 
> 2014-10-02 15:59:08,654 FATAL [IPC Server handler 7 on 46438] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1411750973876_0130_m_000023_1000 - exited : java.io.IOException: java.lang.reflect.InvocationTargetException
> 
>          at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
> 
>          at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
> 
>          at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:302)
> 
>          at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:249)
> 
>          at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileInputFormatShim.getRecordReader(HadoopShimsSecure.java:363)
> 
>          at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:591)
> 
>          at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:168)
> 
>          at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:409)
> 
>          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> 
>          at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> 
>          at java.security.AccessController.doPrivileged(Native Method)
> 
>          at javax.security.auth.Subject.doAs(Subject.java:415)
> 
>          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
> 
>          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> 
> Caused by: java.lang.reflect.InvocationTargetException
> 
>          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> 
>          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> 
>          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 
>          at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> 
>          at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:288)
> 
>          ... 11 more
> 
> Caused by: java.io.IOException: Malformed ORC file hdfs://arnaldo01:8020/apps/hive/warehouse/mds.db/ds_customer_qg/part-m-00001.
Invalid postscript.
> 
>          at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.ensureOrcFooter(ReaderImpl.java:226)
> 
>          at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.extractMetaInfoFromFooter(ReaderImpl.java:336)
> 
>          at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.<init>(ReaderImpl.java:292)
> 
>          at org.apache.hadoop.hive.ql.io.orc.OrcFile.createReader(OrcFile.java:197)
> 
>          at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:999)
> 
>          at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:65)
> 
>          ... 16 more
> 
>  
>  
>  
> Hive Version: 0.13
>  
> Query:
> SELECT   t1.ID_TP_CLIENTE  
>         , t1.COD_CLIENTE  
>         , t1.DESC_MARCA
>         , t1.DOCUMENT_ID_NR             
>         , t1.NM_CLIENTE                
>         , t1.DT_NASCIMENTO           
>         , t1.ID_TP_SEXO     
>         , t1.DT_CRIACAO_CARGA            
>         , t1.DT_LAST_CARGA            
>                                 --
> FROM  ds_customer_qg t1
> join         ds_customer_qg t2       
> on             t1.COD_CLIENTE = t2.COD_CLIENTE
>  
>  
>  
>  
> De: Prasanth Jayachandran [mailto:pjayachandran@hortonworks.com] 
> Enviada em: Wednesday, October 1, 2014 3:24 PM
> Para: user@hive.apache.org
> Assunto: Re: Join error with ORC Hive tables
>  
> Hi
>  
> Can you post the exception stacktrace from hadoop execution logs? What version of hive
are you using?
> Can you provide the join query that you are using?
>  
> Thanks
> Prasanth Jayachandran
>  
> On Oct 1, 2014, at 7:41 AM, Thiago Henrique dos Santos Bento <thiago.bento@b2wdigital.com>
wrote:
> 
> 
> Hi!
>  
> I’m trying to run a simple join with two tables stored as ORC File. (the same query
runs normally without ORC File)
> I got this error while map-reduce job:
> Driver returned: 2.  Errors: OK
> 
> Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
> 
> Total jobs = 1
> 
> Stage-1 is selected by condition resolver.
> 
> Launching Job 1 out of 1
> 
> Number of reduce tasks not specified. Estimated from input data size: 12
> 
> In order to change the average load for a reducer (in bytes):
> 
>   set hive.exec.reducers.bytes.per.reducer=<number>
> 
> In order to limit the maximum number of reducers:
> 
>   set hive.exec.reducers.max=<number>
> 
> In order to set a constant number of reducers:
> 
>   set mapreduce.job.reduces=<number>
> 
> Starting Job = job_1411750973876_0099, Tracking URL = http://arnaldo02:8088/proxy/application_1411750973876_0099/
> 
> Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1411750973876_0099
> 
> Hadoop job information for Stage-1: number of mappers: 42; number of reducers: 12
> 
> 2014-10-01 07:13:51,965 Stage-1 map = 0%,  reduce = 0%
> 
> 2014-10-01 07:14:18,882 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 37.32 sec
> 
> 2014-10-01 07:14:19,990 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU 40.31 sec
> 
> 2014-10-01 07:14:21,100 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU 43.33 sec
> 
> 2014-10-01 07:14:25,544 Stage-1 map = 7%,  reduce = 0%, Cumulative CPU 57.49 sec
> 
> 2014-10-01 07:14:39,996 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU 73.08 sec
> 
> 2014-10-01 07:14:43,338 Stage-1 map = 9%,  reduce = 0%, Cumulative CPU 76.76 sec
> 
> 2014-10-01 07:14:48,882 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU 92.88 sec
> 
> 2014-10-01 07:14:51,107 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU 100.93 sec
> 
> 2014-10-01 07:14:52,212 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU 106.04 sec
> 
> 2014-10-01 07:14:54,426 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 111.02 sec
> 
> 2014-10-01 07:14:57,751 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU 126.41 sec
> 
> 2014-10-01 07:15:05,507 Stage-1 map = 17%,  reduce = 0%, Cumulative CPU 135.71 sec
> 
> 2014-10-01 07:15:10,023 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 146.69 sec
> 
> 2014-10-01 07:15:19,987 Stage-1 map = 19%,  reduce = 0%, Cumulative CPU 156.43 sec
> 
> 2014-10-01 07:15:29,957 Stage-1 map = 19%,  reduce = 1%, Cumulative CPU 168.3 sec
> 
> 2014-10-01 07:15:37,703 Stage-1 map = 21%,  reduce = 1%, Cumulative CPU 179.77 sec
> 
> 2014-10-01 07:15:45,511 Stage-1 map = 23%,  reduce = 1%, Cumulative CPU 192.87 sec
> 
> 2014-10-01 07:15:49,948 Stage-1 map = 25%,  reduce = 1%, Cumulative CPU 204.72 sec
> 
> 2014-10-01 07:15:53,287 Stage-1 map = 26%,  reduce = 1%, Cumulative CPU 211.83 sec
> 
> 2014-10-01 07:16:05,456 Stage-1 map = 28%,  reduce = 1%, Cumulative CPU 231.45 sec
> 
> 2014-10-01 07:16:07,664 Stage-1 map = 29%,  reduce = 1%, Cumulative CPU 233.46 sec
> 
> 2014-10-01 07:16:09,876 Stage-1 map = 29%,  reduce = 2%, Cumulative CPU 233.74 sec
> 
> 2014-10-01 07:16:15,408 Stage-1 map = 32%,  reduce = 2%, Cumulative CPU 255.51 sec
> 
> 2014-10-01 07:16:18,727 Stage-1 map = 33%,  reduce = 2%, Cumulative CPU 258.57 sec
> 
> 2014-10-01 07:16:19,830 Stage-1 map = 33%,  reduce = 3%, Cumulative CPU 261.95 sec
> 
> 2014-10-01 07:16:35,288 Stage-1 map = 35%,  reduce = 3%, Cumulative CPU 288.59 sec
> 
> 2014-10-01 07:16:40,806 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU 298.89 sec
> 
> 2014-10-01 07:16:45,278 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU 306.72 sec
> 
> 2014-10-01 07:16:58,531 Stage-1 map = 38%,  reduce = 4%, Cumulative CPU 312.76 sec
> 
> 2014-10-01 07:17:02,945 Stage-1 map = 40%,  reduce = 4%, Cumulative CPU 321.87 sec
> 
> 2014-10-01 07:17:22,828 Stage-1 map = 42%,  reduce = 4%, Cumulative CPU 336.71 sec
> 
> 2014-10-01 07:17:28,341 Stage-1 map = 43%,  reduce = 5%, Cumulative CPU 343.66 sec
> 
> 2014-10-01 07:17:39,370 Stage-1 map = 44%,  reduce = 5%, Cumulative CPU 350.7 sec
> 
> 2014-10-01 07:17:51,541 Stage-1 map = 47%,  reduce = 5%, Cumulative CPU 365.91 sec
> 
> 2014-10-01 07:18:08,097 Stage-1 map = 49%,  reduce = 5%, Cumulative CPU 375.35 sec
> 
> 2014-10-01 07:18:22,452 Stage-1 map = 49%,  reduce = 7%, Cumulative CPU 383.34 sec
> 
> 2014-10-01 07:18:25,756 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU 383.69 sec
> 
> 2014-10-01 07:19:26,488 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU 389.38 sec
> 
> 2014-10-01 07:19:50,793 Stage-1 map = 52%,  reduce = 7%, Cumulative CPU 391.28 sec
> 
> 2014-10-01 07:19:58,551 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU 386.39 sec
> 
> 2014-10-01 07:20:50,430 Stage-1 map = 52%,  reduce = 7%, Cumulative CPU 393.79 sec
> 
> 2014-10-01 07:21:51,114 Stage-1 map = 52%,  reduce = 7%, Cumulative CPU 411.94 sec
> 
> 2014-10-01 07:22:19,773 Stage-1 map = 54%,  reduce = 7%, Cumulative CPU 417.31 sec
> 
> 2014-10-01 07:23:20,493 Stage-1 map = 54%,  reduce = 7%, Cumulative CPU 423.21 sec
> 
> 2014-10-01 07:24:21,168 Stage-1 map = 54%,  reduce = 7%, Cumulative CPU 425.94 sec
> 
> 2014-10-01 07:25:21,814 Stage-1 map = 54%,  reduce = 7%, Cumulative CPU 441.2 sec
> 
> 2014-10-01 07:25:36,145 Stage-1 map = 55%,  reduce = 7%, Cumulative CPU 442.7 sec
> 
> 2014-10-01 07:26:36,837 Stage-1 map = 55%,  reduce = 7%, Cumulative CPU 448.75 sec
> 
> 2014-10-01 07:26:40,140 Stage-1 map = 55%,  reduce = 8%, Cumulative CPU 449.35 sec
> 
> 2014-10-01 07:27:11,064 Stage-1 map = 56%,  reduce = 8%, Cumulative CPU 455.24 sec
> 
> 2014-10-01 07:27:19,871 Stage-1 map = 57%,  reduce = 8%, Cumulative CPU 460.35 sec
> 
> 2014-10-01 07:27:56,277 Stage-1 map = 58%,  reduce = 8%, Cumulative CPU 481.34 sec
> 
> 2014-10-01 07:28:40,332 Stage-1 map = 59%,  reduce = 8%, Cumulative CPU 504.81 sec
> 
> 2014-10-01 07:29:40,957 Stage-1 map = 59%,  reduce = 8%, Cumulative CPU 510.71 sec
> 
> 2014-10-01 07:30:41,638 Stage-1 map = 59%,  reduce = 8%, Cumulative CPU 513.74 sec
> 
> 2014-10-01 07:31:00,419 Stage-1 map = 60%,  reduce = 8%, Cumulative CPU 514.6 sec
> 
> 2014-10-01 07:32:01,081 Stage-1 map = 60%,  reduce = 8%, Cumulative CPU 518.18 sec
> 
> 2014-10-01 07:32:50,724 Stage-1 map = 62%,  reduce = 8%, Cumulative CPU 520.43 sec
> 
> 2014-10-01 07:33:00,648 Stage-1 map = 60%,  reduce = 8%, Cumulative CPU 521.71 sec
> 
> 2014-10-01 07:33:14,974 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 459.27 sec
> 
> MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
> 
> Ended Job = job_1411750973876_0099 with errors
> 
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> 
> MapReduce Jobs Launched:
> 
> Job 0: Map: 42  Reduce: 12   Cumulative CPU: 459.27 sec   HDFS Read: 1989875843 HDFS
Write: 0 FAIL
> 
> Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
> 
>  
>  
> Any idea?
>  
> Tks,
> Thiago Henrique S. Bento
> B2W Digital / B2W Labs | BIT SP
> thiago.bento@b2wdigital.com
> Tel.: +55 (11) 3454-3024
> .............................................................................................................................................................
> B2W DIGITAL  #Americanas.com  #Shoptime.com  #Submarino.com  #SouBarato.com
>  
>  
> Esta mensagem pode conter informações confidenciais e somente o indivíduo ou entidade
a quem foi destinada pode utilizá-la. A transmissão incorreta da mensagem não acarreta
a perda de sua confidencialidade. Caso esta mensagem tenha sido recebida por engano, solicitamos
que o fato seja comunicado ao remetente e que a mensagem seja eliminada de seu sistema imediatamente.
É vedado a qualquer pessoa que não seja o destinatário usar, revelar, distribuir ou copiar
qualquer parte desta mensagem. Ambiente de comunicação sujeito a monitoramento.
> 
> This message may include confidential information and only the intended addresses have
the right to use it as is, or any part of it. A wrong transmission does not break its confidentiality.
If you've received it because of a mistake or erroneous transmission, please notify the sender
and delete it from your system immediately. This communication environment is controlled and
monitored.
> 
> B2W Digital
>  
> 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to which it
is addressed and may contain information that is confidential, privileged and exempt from
disclosure under applicable law. If the reader of this message is not the intended recipient,
you are hereby notified that any printing, copying, dissemination, distribution, disclosure
or forwarding of this communication is strictly prohibited. If you have received this communication
in error, please contact the sender immediately and delete it from your system. Thank You.


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message