hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bejoy Ks <bejoy...@yahoo.com>
Subject Re: Error while inserting 100GB data in to hive external table
Date Thu, 02 Aug 2012 07:09:57 GMT
Hi Shaik

What does the failed MR task logs say? Can you share the failed Task error log here?

If you are using hadoop 2.0/0.23 then there is a similar issue reported
https://issues.apache.org/jira/browse/HIVE-2804 

 
Regards,
Bejoy KS


________________________________
 From: shaik ahamed <shaik5943@gmail.com>
To: user@hive.apache.org 
Sent: Thursday, August 2, 2012 11:30 AM
Subject: Error while inserting 100GB data in to hive external table
 

Hi Users,
 
 
                  After running for a long time i.e 5hrs i got the below
error
 
Below are the cmds as i used for inseting the data into hive table
 
CREATE EXTERNAL TABLE vender_100GB(vender string,supplier string,order_date string,quantity
int) row format delimited fields terminated by ' ' stored as textfile 
LOCATION '/user/hive/warehouse/vender';
 
CREATE EXTERNAL TABLE vender_100gb_part(supplier string,order_date string,quantity int) PARTITIONED
BY(vender string) row format delimited fields terminated by ' ' 
stored as textfile LOCATION '/user/hive/warehouse/vender';
 

load data local inpath '/usr/local/hadoop_dir/hadoop/big_data/vender_details.txt' overwrite
into table vender_100gb;
 
FROM vender_100GB v INSERT OVERWRITE TABLE vender_100gb_part PARTITION(vender) SELECT v.vender,
v.supplier,v.order_date,v.quantity DISTRIBUTE BY vender;
 
ERROR:
 
2012-08-02 00:17:42,265 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 19901.5 sec
2012-08-02 00:17:43,273 Stage-1 map = 100%,  reduce = 99%, Cumulative CPU 19901.5 sec
2012-08-02 00:17:44,284 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 19901.5 sec
MapReduce Total cumulative CPU time: 0 days 5 hours 31 minutes 41 seconds 500 msec
Ended Job = job_201208011427_0002 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201208011427_0002_m_000414 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000005 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000009 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000012 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000251 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000255 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000028 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000034 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000269 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000044 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000050 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000283 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000060 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000067 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000073 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000303 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000077 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000317 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000084 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000331 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000089 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000345 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000352 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000359 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000102 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000373 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000107 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000387 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000395 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000401 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000409 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000127 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000136 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000141 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000152 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000162 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000172 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000173 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000185 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000195 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000211 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_m_000222 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000000 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000009 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000019 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000029 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000038 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000050 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000059 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000070 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000080 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000090 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000101 (and more) from job job_201208011427_0002
Examining task ID: task_201208011427_0002_r_000110 (and more) from job job_201208011427_0002
Exception in thread "Thread-22" java.lang.RuntimeException: Error while reading from task
log url
        at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(TaskLogProcessor.java:240)
        at org.apache.hadoop.hive.ql.exec.JobDebugger.showJobFailDebugInfo(JobDebugger.java:227)
        at org.apache.hadoop.hive.ql.exec.JobDebugger.run(JobDebugger.java:92)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://md-trngpoc3:50060/tasklog?taskid=attempt_201208011427_0002_r_000078_1&start=-8193
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1436)
        at java.net.URL.openStream(URL.java:1010)
        at org.apache.hadoop.hive.ql.exec.errors.TaskLogProcessor.getStackTraces(TaskLogProcessor.java:192)
        ... 3 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 413  Reduce: 111   Cumulative CPU: 19901.5 sec   HDFS Read: 110754163131
HDFS Write: 291604532 FAIL
Total MapReduce CPU Time Spent: 0 days 5 hours 31 minutes 41 seconds 500 msec
 
Please reply me the route cause for failing this.
 
Thanks in advance
 
shaik
Mime
View raw message