hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From krish <kri22go...@gmail.com>
Subject Re: Fwd: Row exception in Hive while using join
Date Tue, 10 Mar 2015 10:14:11 GMT
hi Swagatika,

*base on further log file analysis i think problem with low disk space.*
*below is full stack trace.*


......
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row (tag=1)
{"key":{"joinkey0":"12"},"value":{"_col2":"."},"alias":1} at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:258) ...
7 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/tmp/hive-root/hive_2015-03-09_10-03-59_970_3646456754594156815-1/_task_tmp.-ext-10001/_tmp.000000_0
could only be replicated to 0 nodes instead of minReplication (=1). There
are 2 datanode(s) running and no node(s) are excluded in this operation.
......

The root cause may be lack of disk space in the HDFS cluster. details of
disk space are

hdfs dfs -df -h

Filesystem Size Used Available Use%

hdfs://x.y.ab.com:8020 159.7 G 21.9 G 110.7 G 14%.

table_line_n_passed having 4767409 rows and 1.1 G size.

similarly table_line_c_passed having 4717082 rows and 1.0 G size .

Does Hive really require that much space (more then available free space
110 G ) to process data. how to calculate how much free space require
before running query .any way to run query within available free space.


i have executed following Hive query

create table table_llv_N_C as select
table_line_n_passed.chromosome_number,table_line_n_passed.position,
table_line_c_passed.id from table_line_n_passed join table_line_c_passed on
(table_line_n_passed.chromosome_number=table_line_c_passed.chromosome_number)

PS: if i used LIMIT 10000 in above query its running fine .

On Mon, Mar 9, 2015 at 9:35 PM, Swagatika Tripathy <swagatikat856@gmail.com>
wrote:

> Hi Krish,
> It seems the data corresponding to that particular row pertaining to key
> 12 is corrupt.Can u try reloading the data and then selecting?
>
> Let me know if it works.
>
> Regards
> Swagatika
> On Mar 5, 2015 4:45 PM, "krish" <kri22gopal@gmail.com> wrote:
>
>>
>> I got the following exception while executing join on Hive Query and
>> reducer hang after 68% completion.
>>
>>
>> java.lang.RuntimeException:
>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
>> processing row (tag=1)
>> {"key":{"joinkey0":"12"},"value":{"_col2":"rs317647905"},"alias":1}
>>         at
>> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:270)
>>         at
>> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:506)
>>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:447)
>>         at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
>>         at java.security.AccessController.doPrivileged(Native Method)
>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>         at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
>>         at org.apache.hadoop.mapred.Child.main(Child.java:262)
>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
>> Error while processing row (tag=1)
>> {"key":{"joinkey0":"12"},"value":{"_col2":"rs317647905"},"alias":1}
>>         at
>> org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:258)
>>         ... 7 more
>> Caused by: org.apache.hadoop.
>>
>> ---------------------------------------------------------------------------------------
>>
>> my query and table structure:
>>
>> create table table_llv_N_C as select
>> table_line_n_passed.chromosome_number,table_line_n_passed.position,
>> table_line_c_passed.id from table_line_n_passed join table_line_c_passed
>> on
>> (table_line_n_passed.chromosome_number=table_line_c_passed.chromosome_number)
>>
>> hive> desc table_line_n_passed;
>> OK
>> chromosome_number       string
>>
>> position        int
>> id      string
>> ref     string
>> alt     string
>> quality double
>> filter  string
>> info    string
>> format  string
>> line6   string
>> Time taken: 0.854 seconds
>> Why am I getting this error, and how can I solve it?
>>
>>
>>
>>
>>
>> --
>> with regards
>> krish!!!!!!
>>
>


-- 
with regards
krish!!!!!!

Mime
View raw message