Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Mon, 21 Aug 2017 16:51:00 +0000 (UTC)
From: "Aleksey Vovchenko (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13095058.1502898651000.80844.1503334260298@Atlassian.JIRA>
In-Reply-To: <JIRA.13095058.1502898651000@Atlassian.JIRA>
References: <JIRA.13095058.1502898651000@Atlassian.JIRA> <JIRA.13095058.1502898651054@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HIVE-17335) Join query with STREAMTABLE fails
 by java.lang.IndexOutOfBoundsException
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 21 Aug 2017 16:51:07 -0000


    [ https://issues.apache.org/jira/browse/HIVE-17335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16135423#comment-16135423 ] 

Aleksey Vovchenko commented on HIVE-17335:
------------------------------------------

[~sershe]
Hi. 
We have this exception because Hive reorders tagOrder and filterOrder without filterMaps reordering. 
But I think filterMaps reordering is not a very good idea. So I have created a patch without changing order of filterMaps.

But now I have a problem. Left join works correctly, but right join returns wrong result. It happens because my fix changes order of processing(filters) rows from a storage. Can you take a look or give me an advice for how I can do it other way?

> Join query with STREAMTABLE fails by java.lang.IndexOutOfBoundsException
> ------------------------------------------------------------------------
>
>                 Key: HIVE-17335
>                 URL: https://issues.apache.org/jira/browse/HIVE-17335
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 1.2.1, 2.1.1
>            Reporter: Aleksey Vovchenko
>            Assignee: Aleksey Vovchenko
>         Attachments: HIVE-17335-branch-2.1.patch
>
>
> Steps to reproduce this issue: 
> h2. STEP 1. Create test tables and insert some data 
> {noformat}
> hive> create table test1(x int, y int, z int);
> hive> create table test2(x int, y int, z int);
> {noformat}
> {noformat}
> hive> insert into table test1 values(1,1,1), (2,2,2);
> hive> insert into table test2 values(1,5,5), (2,6,6);
> {noformat}
> h2. STEP 2. Disable MapJoin
> {noformat}
> hive> set hive.auto.convert.join = false;
> {noformat}
> h2.STEP 3. Run query
> {noformat}
> select /*+ STREAMTABLE(test1) */ test1.*, test2.x from test1 left join test2 on test1.x = test2.x and test1.x > 1;
> {noformat}
> EXPECTED RESULT: 
> {noformat}
> OK
> 1	1	1	NULL
> 2	2	2	2
> {noformat}
> ACTUAL RESULT:
> {noformat} 
> 2017-08-17 00:36:46,305 Stage-1 map = 0%,  reduce = 0%
> 2017-08-17 00:36:51,708 Stage-1 map = 50%,  reduce = 0%, Cumulative CPU 1.25 sec
> 2017-08-17 00:36:52,761 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.35 sec
> 2017-08-17 00:37:17,137 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 2.35 sec
> MapReduce Total cumulative CPU time: 2 seconds 350 msec
> Ended Job = job_1502889241527_0005 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1502889241527_0005_m_000000 (and more) from job job_1502889241527_0005
> Task with the most failures(4): 
> -----
> Task ID:
>   task_1502889241527_0005_r_000000
> -----
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":1},"value":null}
> 	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:257)
> 	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:444)
> 	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> 	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
> 	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":1},"value":null}
> 	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:245)
> 	... 7 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> 	at org.apache.hadoop.hive.ql.exec.JoinOperator.process(JoinOperator.java:138)
> 	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
> 	... 7 more
> Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> 	at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> 	at java.util.ArrayList.get(ArrayList.java:411)
> 	at org.apache.hadoop.hive.ql.exec.JoinUtil.isFiltered(JoinUtil.java:248)
> 	at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.getFilteredValue(CommonJoinOperator.java:420)
> 	at org.apache.hadoop.hive.ql.exec.JoinOperator.process(JoinOperator.java:91)
> 	... 8 more
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched: 
> {noformat} 


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)