hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikram Dixit K (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4650) Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x
Date Thu, 06 Jun 2013 22:44:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13677618#comment-13677618
] 

Vikram Dixit K commented on HIVE-4650:
--------------------------------------

This is related to the issue HIVE-4502. I have verified that the plans generated for this
query with the existing patch on HIVE-4502 look fine. I will close this based on the resolution
of that issue.
                
> Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-4650
>                 URL: https://issues.apache.org/jira/browse/HIVE-4650
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.11.0
>         Environment: HortonWorks 1.3 distro on x86_64 Centos 6 
>            Reporter: Bruce Nelson
>
> working from a simple table in Hive 
> hive> desc cmnt
>     > ;
> OK
> x1                      int                     None                
> x2                      int                     None                
> x3                      int                     None                
> x4                      int                     None                
> y                       double                  None             
> hive> select * from cmnt;
> OK
> 7       26      6       60      78.5
> 1       29      15      52      74.3
> 11      56      8       20      104.3
> 11      31      8       47      87.6
> 7       52      6       33      95.9
> 11      55      9       22      109.2
> 3       71      17      6       102.7
> 1       31      22      44      72.5
> 2       54      18      22      93.1
> 21      47      4       26      115.9
> 1       40      23      34      83.8
> 11      66      9       12      113.3
> 10      68      8       12      109.4
> A query that joins and transforms against this table : 
> select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from ( select
/*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) * 1/(sd2) VAL002,(VAL003-
mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4) VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from (
select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3
) v3 join (select count(*) c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003) mu3,avg(VAL004)
mu4,avg(VAL005) mu5, stddev_pop(VAL001) sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004)
sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4
VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7) obj1_6 ;
> Generates during Stage-3 : 
> setting HADOOP_USER_NAME        test
> Execution log at: /tmp/test/.log
> 2013-06-03 12:40:55     Starting to launch local task to process map join;      maximum
memory = 1065484288
> 2013-06-03 12:40:56     Processing rows:        1       Hashtable size: 1       Memory
usage:   7175528   rate:   0.007
> 2013-06-03 12:40:56     Dump the hashtable into file: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
> 2013-06-03 12:40:56     Upload 1 File to: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable
File size: 334
> 2013-06-03 12:40:56     End of local task; Time Taken: 0.726 sec.
> Execution completed successfully
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Mapred Local Task Succeeded . Convert the Join into MapJoin
> Launching Job 2 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201306022123_0045, Tracking URL = http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job  -kill job_201306022123_0045
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0
> 2013-06-03 00:41:05,895 Stage-3 map = 0%,  reduce = 0%
> 2013-06-03 00:41:40,687 Stage-3 map = 100%,  reduce = 100%
> Ended Job = job_201306022123_0045 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL: http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045
> Examining task ID: task_201306022123_0045_m_000002 (and more) from job job_201306022123_0045
> Task with the most failures(4): 
> -----
> Task ID:
>   task_201306022123_0045_m_000000
> URL:
>   http://sun1vm3:50030/taskdetails.jsp?jobid=job_201306022123_0045&tipid=task_201306022123_0045_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365)
>         at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>         at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)
>         at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377)
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
>         at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
>         at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611)
>         at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
>         ... 8 more
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186)
>         ... 14 more
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
> If "hive.auto.convert.join = false" is set then the all the query stages work OK. The
same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin working.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message