hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Jarcec Cecho <jar...@apache.org>
Subject Re: Hive 0.7.1 Query hands
Date Thu, 21 Feb 2013 15:59:21 GMT
Hi sir,
the root cause of your issues seems to be java.io.EOFException, that based on the java doc
description means the following:

  "Signals that an end of file or end of stream has been reached unexpectedly during input."

What is the health status of the box with ip 10.6.0.55? Isn't it by any chance having some
issues? Port 8021 is used for TaskTracker, so I would start by connecting to that box and
checking the TT logs.

Jarcec

On Thu, Feb 21, 2013 at 02:44:09PM +0300, Павел Мезенцев wrote:
> Hello!
> 
> I use Hive 0.7.1 over Hadoop 0.20.2 (CHD3u3) on 70 nodes cluster.
> I have a trouble with query like this:
> 
> *FROM* (  *SELECT* *id*, {expressions} *FROM* table1   *WHERE*
> day='2013-02-16' *AND* ({conditions1})*UNION* *ALL*  *SELECT* *id*,
> {expressions} *FROM* table2   *WHERE* day='2013-02-16' *AND*
> (conditions)*UNION* *ALL*  *SELECT* *id*, {expressions} *FROM* table3
> *WHERE* day='2013-02-16' *AND* (conditions)*UNION* *ALL*  *SELECT*
> *id*, {expressions} *FROM* table4  *WHERE* day='2013-02-16' *AND*
> (conditions)) union_tmp*INSERT* OVERWRITE *table* result_table
> *PARTITION* (day='2013-02-16')*SELECT* *id*, transformations
> (expressions)*GROUP* *BY* *id*;
> 
> it had 4865 map tasks and 100 reduce tasks.
> first 4780 map taks completed succefull and last
> 85 tasks hangs.
> 
> All this tasks hands with no progress,
> 
> 
> and no tasks attempts for each:
> 
> One hour later after hang situation starting, job failed with exeption:
> 
> 2013-02-20 20:02:02,000 Stage-1 map = 0%,  reduce = 0%
> 2013-02-20 20:02:40,679 Stage-1 map = 1%,  reduce = 0%
> 2013-02-20 20:02:54,022 Stage-1 map = 2%,  reduce = 0%
> 2013-02-20 20:03:14,129 Stage-1 map = 3%,  reduce = 0%
> ......................................................
> 2013-02-20 21:18:00,361 Stage-1 map = 98%,  reduce = 22%
> 2013-02-20 21:18:05,691 Stage-1 map = 98%,  reduce = 23%
> java.io.IOException: Call to statlabjt/10.6.0.55:8021 failed on local
> exception: java.io.EOFException
> 	at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
> 	at org.apache.hadoop.ipc.Client.call(Client.java:1110)
> 	at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
> 	at org.apache.hadoop.mapred.$Proxy8.getJobStatus(Unknown Source)
> 	at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:1053)
> 	at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:1065)
> 	at org.apache.hadoop.hive.ql.exec.ExecDriver.progress(ExecDriver.java:351)
> 	at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:672)
> 	at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
> 	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
> 	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> 	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
> 	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:425)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(DataInputStream.java:375)
> 	at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:815)
> 	at org.apache.hadoop.ipc.Client$Connection.run(Client.java:724)
> Ended Job = job_201302152355_4764 with exception
> 'java.io.IOException(Call to statlabjt/10.6.0.55:8021 failed on local
> exception: java.io.EOFException)'
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> 
> How to find reasons of such situation?
> How to prevent such situations in future?
> 
> Best regards
> Mezentsev Pavel

Mime
View raw message