hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saumil Mayani (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-16791) Tez engine giving very inaccurate results on SMB Map joins while map-join and shuffle join gets correct results
Date Tue, 30 May 2017 20:51:04 GMT

     [ https://issues.apache.org/jira/browse/HIVE-16791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Saumil Mayani updated HIVE-16791:
---------------------------------
    Description: 
SMB Join gives incorrect results. 
{code}
SMB-Join
set hive.execution.engine=tez;
set hive.enforce.sortmergebucketmapjoin=false;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.auto.convert.sortmerge.join=true;
set hive.auto.convert.join=true;
set hive.auto.convert.join.noconditionaltask.size=500000;

OK
2016	1	11999639
2016	2	18955110
2017	2	22217437
Time taken: 92.647 seconds, Fetched: 3 row(s)
{code}
{code}
MAP-JOIN
set hive.execution.engine=tez;
set hive.enforce.sortmergebucketmapjoin=false;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.auto.convert.sortmerge.join=true;
set hive.auto.convert.join=true;
set hive.auto.convert.join.noconditionaltask.size=50000000;
OK
2016	1	26586093
2016	2	17724062
2017	2	8862031
Time taken: 17.49 seconds, Fetched: 3 row(s)
{code}
{code}
Shuffle Join
set hive.execution.engine=tez;
set hive.enforce.sortmergebucketmapjoin=false;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.auto.convert.sortmerge.join=false;
set hive.auto.convert.join=false;
set hive.auto.convert.join.noconditionaltask.size=50000000;

OK
2016	1	26586093
2016	2	17724062
2017	2	8862031
Time taken: 38.575 seconds, Fetched: 3 row(s)
{code}

  was:
SMB Join gives incorrect results.
{code}
SMB-Join
set hive.execution.engine=tez;
set hive.enforce.sortmergebucketmapjoin=false;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.auto.convert.sortmerge.join=true;
set hive.auto.convert.join=true;
set hive.auto.convert.join.noconditionaltask.size=500000;

OK
2016	1	11999639
2016	2	18955110
2017	2	22217437
Time taken: 92.647 seconds, Fetched: 3 row(s)
{code}
{code}
MAP-JOIN
set hive.execution.engine=tez;
set hive.enforce.sortmergebucketmapjoin=false;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.auto.convert.sortmerge.join=true;
set hive.auto.convert.join=true;
set hive.auto.convert.join.noconditionaltask.size=50000000;
OK
2016	1	26586093
2016	2	17724062
2017	2	8862031
Time taken: 17.49 seconds, Fetched: 3 row(s)
{code}
{code}
Shuffle Join
set hive.execution.engine=tez;
set hive.enforce.sortmergebucketmapjoin=false;
set hive.optimize.bucketmapjoin=true;
set hive.optimize.bucketmapjoin.sortedmerge=true;
set hive.auto.convert.sortmerge.join=false;
set hive.auto.convert.join=false;
set hive.auto.convert.join.noconditionaltask.size=50000000;

OK
2016	1	26586093
2016	2	17724062
2017	2	8862031
Time taken: 38.575 seconds, Fetched: 3 row(s)
{code}


> Tez engine giving very inaccurate results on SMB Map joins while map-join and shuffle
join gets correct results
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16791
>                 URL: https://issues.apache.org/jira/browse/HIVE-16791
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive, HiveServer2
>            Reporter: Saumil Mayani
>            Assignee: Deepak Jaiswal
>
> SMB Join gives incorrect results. 
> {code}
> SMB-Join
> set hive.execution.engine=tez;
> set hive.enforce.sortmergebucketmapjoin=false;
> set hive.optimize.bucketmapjoin=true;
> set hive.optimize.bucketmapjoin.sortedmerge=true;
> set hive.auto.convert.sortmerge.join=true;
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.noconditionaltask.size=500000;
> OK
> 2016	1	11999639
> 2016	2	18955110
> 2017	2	22217437
> Time taken: 92.647 seconds, Fetched: 3 row(s)
> {code}
> {code}
> MAP-JOIN
> set hive.execution.engine=tez;
> set hive.enforce.sortmergebucketmapjoin=false;
> set hive.optimize.bucketmapjoin=true;
> set hive.optimize.bucketmapjoin.sortedmerge=true;
> set hive.auto.convert.sortmerge.join=true;
> set hive.auto.convert.join=true;
> set hive.auto.convert.join.noconditionaltask.size=50000000;
> OK
> 2016	1	26586093
> 2016	2	17724062
> 2017	2	8862031
> Time taken: 17.49 seconds, Fetched: 3 row(s)
> {code}
> {code}
> Shuffle Join
> set hive.execution.engine=tez;
> set hive.enforce.sortmergebucketmapjoin=false;
> set hive.optimize.bucketmapjoin=true;
> set hive.optimize.bucketmapjoin.sortedmerge=true;
> set hive.auto.convert.sortmerge.join=false;
> set hive.auto.convert.join=false;
> set hive.auto.convert.join.noconditionaltask.size=50000000;
> OK
> 2016	1	26586093
> 2016	2	17724062
> 2017	2	8862031
> Time taken: 38.575 seconds, Fetched: 3 row(s)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message