hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Haviv <daniel.ha...@veracity-group.com>
Subject Re: Disabling local mode optimization
Date Mon, 02 Nov 2015 09:16:55 GMT
Hi,
I'm trying to set  hive.exec.mode.local.auto.inputbytes.max &
hive.exec.mode.local.auto.tasks.max to 1 or 0 but still local mode is being
used instead of M/R.

Any ideas?

Thank you.
Daniel

On Thu, Sep 3, 2015 at 8:02 AM, sreebalineni . <sreebalineni@gmail.com>
wrote:

> Hi,
>
> Is not it that you should set it true, by default it is disabled which is
> false.
>
> Hive analyzes the size of each map-reduce job in a query and may run it
> locally if the following thresholds are satisfied:
>
>    - The total input size of the job is lower than:
>    hive.exec.mode.local.auto.inputbytes.max (128MB by default)
>    - The total number of map-tasks is less than:
>    hive.exec.mode.local.auto.tasks.max (4 by default)
>    - The total number of reduce tasks required is 1 or 0.
>
> So for queries over small data sets, or for queries with multiple
> map-reduce jobs where the input to subsequent jobs is substantially smaller
> (because of reduction/filtering in the prior job), jobs may be run locally.
>
> so we may need to check the sizeof your input, which version of hive are
> you using? it can work only from Hive 0.7 onwards
>
> On Wed, Sep 2, 2015 at 4:46 PM, Daniel Haviv <
> daniel.haviv@veracity-group.com> wrote:
>
>> Hi,
>> I would like to disable the optimization where a query that just selects
>> data is running without mapreduce (local mode).
>>
>> hive.exec.mode.local.auto is set to false but hive still runs in local mode for some
queries.
>>
>>
>> How can I disable local mode completely?
>>
>>
>> Thank you.
>>
>> Daniel
>>
>>
>

Mime
View raw message