hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nagarjuna Vissarapu <nagarjuna.v...@gmail.com>
Subject Re: Join error with ORC Hive tables
Date Wed, 01 Oct 2014 18:27:59 GMT
Set -m flag to 1 programmatically
On Oct 1, 2014 8:13 PM, "Thiago Henrique dos Santos Bento" <
thiago.bento@b2wdigital.com> wrote:

>  Hi!
>
>
>
> I'm trying to run a simple join with two tables stored as ORC File. (the
> same query runs normally without ORC File)
>
> I got this error while map-reduce job:
>
> Driver returned: 2.  Errors: OK
>
> Query ID = hue_20141001071313_33e31d68-3b4a-49d5-acc0-30c0a42276ab
>
> Total jobs = 1
>
> Stage-1 is selected by condition resolver.
>
> Launching Job 1 out of 1
>
> Number of reduce tasks not specified. Estimated from input data size: 12
>
> In order to change the average load for a reducer (in bytes):
>
>   set hive.exec.reducers.bytes.per.reducer=<number>
>
> In order to limit the maximum number of reducers:
>
>   set hive.exec.reducers.max=<number>
>
> In order to set a constant number of reducers:
>
>   set mapreduce.job.reduces=<number>
>
> Starting Job = job_1411750973876_0099, Tracking URL =
> http://arnaldo02:8088/proxy/application_1411750973876_0099/
>
> Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill job_1411750973876_0099
>
> Hadoop job information for Stage-1: number of mappers: 42; number of
> reducers: 12
>
> 2014-10-01 07:13:51,965 Stage-1 map = 0%,  reduce = 0%
>
> 2014-10-01 07:14:18,882 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU
> 37.32 sec
>
> 2014-10-01 07:14:19,990 Stage-1 map = 3%,  reduce = 0%, Cumulative CPU
> 40.31 sec
>
> 2014-10-01 07:14:21,100 Stage-1 map = 5%,  reduce = 0%, Cumulative CPU
> 43.33 sec
>
> 2014-10-01 07:14:25,544 Stage-1 map = 7%,  reduce = 0%, Cumulative CPU
> 57.49 sec
>
> 2014-10-01 07:14:39,996 Stage-1 map = 8%,  reduce = 0%, Cumulative CPU
> 73.08 sec
>
> 2014-10-01 07:14:43,338 Stage-1 map = 9%,  reduce = 0%, Cumulative CPU
> 76.76 sec
>
> 2014-10-01 07:14:48,882 Stage-1 map = 11%,  reduce = 0%, Cumulative CPU
> 92.88 sec
>
> 2014-10-01 07:14:51,107 Stage-1 map = 12%,  reduce = 0%, Cumulative CPU
> 100.93 sec
>
> 2014-10-01 07:14:52,212 Stage-1 map = 13%,  reduce = 0%, Cumulative CPU
> 106.04 sec
>
> 2014-10-01 07:14:54,426 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU
> 111.02 sec
>
> 2014-10-01 07:14:57,751 Stage-1 map = 15%,  reduce = 0%, Cumulative CPU
> 126.41 sec
>
> 2014-10-01 07:15:05,507 Stage-1 map = 17%,  reduce = 0%, Cumulative CPU
> 135.71 sec
>
> 2014-10-01 07:15:10,023 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU
> 146.69 sec
>
> 2014-10-01 07:15:19,987 Stage-1 map = 19%,  reduce = 0%, Cumulative CPU
> 156.43 sec
>
> 2014-10-01 07:15:29,957 Stage-1 map = 19%,  reduce = 1%, Cumulative CPU
> 168.3 sec
>
> 2014-10-01 07:15:37,703 Stage-1 map = 21%,  reduce = 1%, Cumulative CPU
> 179.77 sec
>
> 2014-10-01 07:15:45,511 Stage-1 map = 23%,  reduce = 1%, Cumulative CPU
> 192.87 sec
>
> 2014-10-01 07:15:49,948 Stage-1 map = 25%,  reduce = 1%, Cumulative CPU
> 204.72 sec
>
> 2014-10-01 07:15:53,287 Stage-1 map = 26%,  reduce = 1%, Cumulative CPU
> 211.83 sec
>
> 2014-10-01 07:16:05,456 Stage-1 map = 28%,  reduce = 1%, Cumulative CPU
> 231.45 sec
>
> 2014-10-01 07:16:07,664 Stage-1 map = 29%,  reduce = 1%, Cumulative CPU
> 233.46 sec
>
> 2014-10-01 07:16:09,876 Stage-1 map = 29%,  reduce = 2%, Cumulative CPU
> 233.74 sec
>
> 2014-10-01 07:16:15,408 Stage-1 map = 32%,  reduce = 2%, Cumulative CPU
> 255.51 sec
>
> 2014-10-01 07:16:18,727 Stage-1 map = 33%,  reduce = 2%, Cumulative CPU
> 258.57 sec
>
> 2014-10-01 07:16:19,830 Stage-1 map = 33%,  reduce = 3%, Cumulative CPU
> 261.95 sec
>
> 2014-10-01 07:16:35,288 Stage-1 map = 35%,  reduce = 3%, Cumulative CPU
> 288.59 sec
>
> 2014-10-01 07:16:40,806 Stage-1 map = 37%,  reduce = 3%, Cumulative CPU
> 298.89 sec
>
> 2014-10-01 07:16:45,278 Stage-1 map = 38%,  reduce = 3%, Cumulative CPU
> 306.72 sec
>
> 2014-10-01 07:16:58,531 Stage-1 map = 38%,  reduce = 4%, Cumulative CPU
> 312.76 sec
>
> 2014-10-01 07:17:02,945 Stage-1 map = 40%,  reduce = 4%, Cumulative CPU
> 321.87 sec
>
> 2014-10-01 07:17:22,828 Stage-1 map = 42%,  reduce = 4%, Cumulative CPU
> 336.71 sec
>
> 2014-10-01 07:17:28,341 Stage-1 map = 43%,  reduce = 5%, Cumulative CPU
> 343.66 sec
>
> 2014-10-01 07:17:39,370 Stage-1 map = 44%,  reduce = 5%, Cumulative CPU
> 350.7 sec
>
> 2014-10-01 07:17:51,541 Stage-1 map = 47%,  reduce = 5%, Cumulative CPU
> 365.91 sec
>
> 2014-10-01 07:18:08,097 Stage-1 map = 49%,  reduce = 5%, Cumulative CPU
> 375.35 sec
>
> 2014-10-01 07:18:22,452 Stage-1 map = 49%,  reduce = 7%, Cumulative CPU
> 383.34 sec
>
> 2014-10-01 07:18:25,756 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU
> 383.69 sec
>
> 2014-10-01 07:19:26,488 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU
> 389.38 sec
>
> 2014-10-01 07:19:50,793 Stage-1 map = 52%,  reduce = 7%, Cumulative CPU
> 391.28 sec
>
> 2014-10-01 07:19:58,551 Stage-1 map = 50%,  reduce = 7%, Cumulative CPU
> 386.39 sec
>
> 2014-10-01 07:20:50,430 Stage-1 map = 52%,  reduce = 7%, Cumulative CPU
> 393.79 sec
>
> 2014-10-01 07:21:51,114 Stage-1 map = 52%,  reduce = 7%, Cumulative CPU
> 411.94 sec
>
> 2014-10-01 07:22:19,773 Stage-1 map = 54%,  reduce = 7%, Cumulative CPU
> 417.31 sec
>
> 2014-10-01 07:23:20,493 Stage-1 map = 54%,  reduce = 7%, Cumulative CPU
> 423.21 sec
>
> 2014-10-01 07:24:21,168 Stage-1 map = 54%,  reduce = 7%, Cumulative CPU
> 425.94 sec
>
> 2014-10-01 07:25:21,814 Stage-1 map = 54%,  reduce = 7%, Cumulative CPU
> 441.2 sec
>
> 2014-10-01 07:25:36,145 Stage-1 map = 55%,  reduce = 7%, Cumulative CPU
> 442.7 sec
>
> 2014-10-01 07:26:36,837 Stage-1 map = 55%,  reduce = 7%, Cumulative CPU
> 448.75 sec
>
> 2014-10-01 07:26:40,140 Stage-1 map = 55%,  reduce = 8%, Cumulative CPU
> 449.35 sec
>
> 2014-10-01 07:27:11,064 Stage-1 map = 56%,  reduce = 8%, Cumulative CPU
> 455.24 sec
>
> 2014-10-01 07:27:19,871 Stage-1 map = 57%,  reduce = 8%, Cumulative CPU
> 460.35 sec
>
> 2014-10-01 07:27:56,277 Stage-1 map = 58%,  reduce = 8%, Cumulative CPU
> 481.34 sec
>
> 2014-10-01 07:28:40,332 Stage-1 map = 59%,  reduce = 8%, Cumulative CPU
> 504.81 sec
>
> 2014-10-01 07:29:40,957 Stage-1 map = 59%,  reduce = 8%, Cumulative CPU
> 510.71 sec
>
> 2014-10-01 07:30:41,638 Stage-1 map = 59%,  reduce = 8%, Cumulative CPU
> 513.74 sec
>
> 2014-10-01 07:31:00,419 Stage-1 map = 60%,  reduce = 8%, Cumulative CPU
> 514.6 sec
>
> 2014-10-01 07:32:01,081 Stage-1 map = 60%,  reduce = 8%, Cumulative CPU
> 518.18 sec
>
> 2014-10-01 07:32:50,724 Stage-1 map = 62%,  reduce = 8%, Cumulative CPU
> 520.43 sec
>
> 2014-10-01 07:33:00,648 Stage-1 map = 60%,  reduce = 8%, Cumulative CPU
> 521.71 sec
>
> 2014-10-01 07:33:14,974 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU
> 459.27 sec
>
> MapReduce Total cumulative CPU time: 7 minutes 39 seconds 270 msec
>
> Ended Job = job_1411750973876_0099 with errors
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
>
> MapReduce Jobs Launched:
>
> Job 0: Map: 42  Reduce: 12   Cumulative CPU: 459.27 sec   HDFS Read:
> 1989875843 HDFS Write: 0 FAIL
>
> Total MapReduce CPU Time Spent: 7 minutes 39 seconds 270 msec
>
>
>
>
>
> Any idea?
>
>
>
> Tks,
>
> *Thiago Henrique S. Bento*
>
> B2W Digital / B2W Labs | BIT SP
>
> thiago.bento@b2wdigital.com
>
> Tel.: +55 (11) 3454-3024
>
>
> .............................................................................................................................................................
>
> *B**2**W DIGITAL*  #Americanas.com <http://americanas.com/>  #Shoptime.com
> <http://shoptime.com/>  #Submarino.com <http://submarino.com/>  #
> SouBarato.com <http://soubarato.com/>
>
>
>
>
>  ------------------------------
>
> Esta mensagem pode conter informações confidenciais e somente o indivíduo
> ou entidade a quem foi destinada pode utilizá-la. A transmissão incorreta
> da mensagem não acarreta a perda de sua confidencialidade. Caso esta
> mensagem tenha sido recebida por engano, solicitamos que o fato seja
> comunicado ao remetente e que a mensagem seja eliminada de seu sistema
> imediatamente. É vedado a qualquer pessoa que não seja o destinatário usar,
> revelar, distribuir ou copiar qualquer parte desta mensagem. Ambiente de
> comunicação sujeito a monitoramento.
>
> This message may include confidential information and only the intended
> addresses have the right to use it as is, or any part of it. A wrong
> transmission does not break its confidentiality. If you've received it
> because of a mistake or erroneous transmission, please notify the sender
> and delete it from your system immediately. This communication environment
> is controlled and monitored.
>
> B2W Digital
>
>
>

Mime
View raw message