hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Slacum <wsla...@gmail.com>
Subject Re: Origin of hive.auto.convert.sortmerge.join.noconditionaltask
Date Wed, 05 Aug 2015 13:33:51 GMT
Makes me feel not insane :) Thanks!



> On Aug 5, 2015, at 12:30 AM, Lefty Leverenz <leftyleverenz@gmail.com> wrote:
> 
> Good question.  I can't find it in any Hive releases.  There's hive.auto.convert.join.noconditionaltask
(starting in 0.11.0) but not hive.auto.convert.sortmerge.join.noconditionaltask.
> 
> Several JIRA issues mention it, including the 0.13.0 release note for HIVE-6098 "Merge
Tez branch into trunk":
> 
>> Hive settings: 
>> 
>> // needed because SMB isn't supported on tez yet 
>> set hive.optimize.bucketmapjoin=false; 
>> set hive.optimize.bucketmapjoin.sortedmerge=false; 
>> set hive.auto.convert.sortmerge.join=false; 
>> set hive.auto.convert.sortmerge.join.noconditionaltask=false; 
>> set hive.auto.convert.join.noconditionaltask=true; 
> 
> And it was in the Join Optimization wikidoc from July 2013 until last month, when Vikram
removed it (see page history): 
>> Auto Conversion to SMB Map Join
>> Sort-Merge-Bucket (SMB) joins can be converted to SMB map joins as well. SMB joins
are used wherever the tables are sorted and bucketed. The join boils down to just merging
the already sorted tables, allowing this operation to be faster than an ordinary map-join.
However, if the tables are partitioned, there could be a slow down as each mapper would need
to get a very small chunk of a partition which has a single key.
>> The following configuration settings enable the conversion of an SMB to a map-join
SMB:
>> set hive.auto.convert.sortmerge.join=true;
>> set hive.optimize.bucketmapjoin = true;
>> set hive.optimize.bucketmapjoin.sortedmerge = true;
>> set hive.auto.convert.sortmerge.join.noconditionaltask=true;
> 
> -- Lefty
> 
> 
> 
>> On Tue, Aug 4, 2015 at 4:46 PM, William Slacum <wslacum@gmail.com> wrote:
>> Hi all,
>> 
>> I've had some questions from users regarding setting `hive.auto.convert.sortmerge.join.noconditionaltask`.
I see, in some documentation from users and vendors, that it is recommended to set this parameter.
In neither Hive 0.12 nor 0.14 can I find in HiveConf where this is actually defined and used.
Am I correct in thinking that this is just some cruft that's survived without verification?
>> 
>> Thanks!
> 

Mime
View raw message