hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vighnesh Avadhani (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-3640) Reducer allocation is incorrect if enforce bucketing and mapred.reduce.tasks are both set
Date Thu, 01 Nov 2012 17:35:12 GMT

     [ https://issues.apache.org/jira/browse/HIVE-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vighnesh Avadhani updated HIVE-3640:
------------------------------------

    Status: Patch Available  (was: Open)
    
> Reducer allocation is incorrect if enforce bucketing and mapred.reduce.tasks are both
set
> -----------------------------------------------------------------------------------------
>
>                 Key: HIVE-3640
>                 URL: https://issues.apache.org/jira/browse/HIVE-3640
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Vighnesh Avadhani
>            Assignee: Vighnesh Avadhani
>            Priority: Minor
>         Attachments: HIVE-3640.1.patch.txt
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> When I enforce bucketing and fix the number of reducers via mapred.reduce.tasks Hive
ignores my input and instead takes the largest value <= hive.exec.reducers.max that is
also an even divisor of num_buckets. In other words, if I set 1024 buckets and set mapred.reduce.tasks=1024
I'll get. . . 256 reducers. If I set 1997 buckets and set mapred.reduce.tasks=1997 I'll get.
. . 1 reducer. 
> This is totally crazy, and it's far, far crazier when the data inputs get large. In the
latter case the bucketing job will almost certainly fail because we'll most likely try to
stuff several TB of input through a single reducer. We'll also drastically reduce the effectiveness
of bucketing, since the buckets themselves will be larger.
> If the user sets mapred.reduce.tasks in a query that inserts into a bucketed table we
should either accept that value or raise an exception if it's invalid relative to the number
of buckets. We should absolutely NOT override the user's direction and fall back on automatically
allocating reducers based on some obscure logic dictated by completely different setting.

> I have yet to encounter a single person who expected this the first time, so it's clearly
a bug.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message