hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yongzhi Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-20915) Make dynamic sort partition optimization available to HoS and MR
Date Fri, 07 Dec 2018 16:29:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-20915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16713058#comment-16713058
] 

Yongzhi Chen commented on HIVE-20915:
-------------------------------------

[~kgyrtkirk], the optimizer uses one reducer to sort the output and put it into one file (one
for each partition, so no need for group by). The behavior is always like this. You can check
old patch(not-checked in) which try to enable the optimizer by default, enable the optimizer
will remove the auto gather thing. 
https://issues.apache.org/jira/secure/attachment/12898098/HIVE-17935.7.patch#file-5


> Make dynamic sort partition optimization available to HoS and MR
> ----------------------------------------------------------------
>
>                 Key: HIVE-20915
>                 URL: https://issues.apache.org/jira/browse/HIVE-20915
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Planning
>    Affects Versions: 4.0.0
>            Reporter: Yongzhi Chen
>            Assignee: Yongzhi Chen
>            Priority: Major
>             Fix For: 4.0.0
>
>         Attachments: HIVE-20915.1.patch, HIVE-20915.2.patch, HIVE-20915.3.patch, HIVE-20915.4.patch,
HIVE-20915.5.patch
>
>
> HIVE-20703 put dynamic sort partition optimization under cost based decision, but it
also makes the optimizer only available to tez. hive.optimize.sort.dynamic.partition works
with other execution engines for a long time, we should keep the optimizer available to them.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message