hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4650) Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x
Date Tue, 23 Jul 2013 03:54:52 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13716019#comment-13716019
] 

Ashutosh Chauhan commented on HIVE-4650:
----------------------------------------

Bruce,
I will encourage you to ask this question on Hortonworks mailing list which is the best place
to ask questions about Hortonworks distribution. 
The next apache release of Hive will definitely contain this fix.
                
> Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-4650
>                 URL: https://issues.apache.org/jira/browse/HIVE-4650
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.11.0
>         Environment: HortonWorks 1.3 distro on x86_64 Centos 6 
>            Reporter: Bruce Nelson
>             Fix For: 0.12.0
>
>         Attachments: join32_lessSize.q.out
>
>
> working from a simple table in Hive 
> hive> desc cmnt
>     > ;
> OK
> x1                      int                     None                
> x2                      int                     None                
> x3                      int                     None                
> x4                      int                     None                
> y                       double                  None             
> hive> select * from cmnt;
> OK
> 7       26      6       60      78.5
> 1       29      15      52      74.3
> 11      56      8       20      104.3
> 11      31      8       47      87.6
> 7       52      6       33      95.9
> 11      55      9       22      109.2
> 3       71      17      6       102.7
> 1       31      22      44      72.5
> 2       54      18      22      93.1
> 21      47      4       26      115.9
> 1       40      23      34      83.8
> 11      66      9       12      113.3
> 10      68      8       12      109.4
> A query that joins and transforms against this table : 
> select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from ( select
/*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) * 1/(sd2) VAL002,(VAL003-
mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4) VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from (
select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3
) v3 join (select count(*) c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003) mu3,avg(VAL004)
mu4,avg(VAL005) mu5, stddev_pop(VAL001) sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004)
sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4
VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7) obj1_6 ;
> Generates during Stage-3 : 
> setting HADOOP_USER_NAME        test
> Execution log at: /tmp/test/.log
> 2013-06-03 12:40:55     Starting to launch local task to process map join;      maximum
memory = 1065484288
> 2013-06-03 12:40:56     Processing rows:        1       Hashtable size: 1       Memory
usage:   7175528   rate:   0.007
> 2013-06-03 12:40:56     Dump the hashtable into file: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2013-06-03 12:40:56     Upload 1 File to: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
File size: 334
> 2013-06-03 12:40:56     End of local task; Time Taken: 0.726 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 2 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201306022123_0045, Tracking URL = http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job  -kill job_201306022123_0045
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
> 2013-06-03 00:41:05,895 Stage-3 map = 0%,  reduce = 0%
> 2013-06-03 00:41:40,687 Stage-3 map = 100%,  reduce = 100%
> Ended Job = job_201306022123_0045 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL: http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Examining task ID: task_201306022123_0045_m_000002 (and more) from job job_201306022123_0045
> Task with the most failures(4): 
> -----
> Task ID:
>   task_201306022123_0045_m_000000
> URL:
>   http://sun1vm3:50030/taskdetails.jsp?jobid=job_201306022123_0045&tipid=task_201306022123_0045_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
>         at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
>         at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
>         at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
>         ... 8 more
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
>         ... 14 more
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
> If "hive.auto.convert.join = false" is set then the all the query stages work OK. The
same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin working.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message