hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joydeep Sen Sarma <jssa...@facebook.com>
Subject RE: Hive local mode on by default for 0.6.0
Date Tue, 10 Aug 2010 04:09:25 GMT
> I personally feel like it would be better to turn this off.

i agree. unless there's contrary opinion - i am just going to turn it off by default  in with
some upcoming diff/checkin (and reply to this thread again)

(just realized that the title of this thread is misleading ..)
_______________________________________
From: Edward Capriolo [edlinuxguru@gmail.com]
Sent: Monday, August 09, 2010 8:04 PM
To: hive-dev@hadoop.apache.org
Subject: Re: Hive local mode on by default for 0.6.0

On Mon, Aug 9, 2010 at 9:28 PM, Joydeep Sen Sarma <jssarma@facebook.com> wrote:
> We enabled a feature called 'auto-local mode' (hive-1408). The query processor looks
at the size of the input and decides dynamically whether local mode execution can be done.
The determination is done on a per job level for a multi-job query.
>
> We enabled it by default in trunk so it can get some coverage. Local mode support in
0.6 has some bugs (in fact a big part of this jira was a comprehensive test for local mode
and small fixes for the bugs that this uncovered). The relevant option is:
>
> set hive.exec.mode.local.auto=<true/false>
>
>
> I have been a little worried about enabling this by default - we can turn it off if required.
The case that worries me the most is if a lot of users refer to scripts (via transform clauses)
that are only available in the cluster nodes and not in the client node. Another assumption
is that mapred.local.dir is set to a value valid on the client side (which may not be the
case if the same hadoop config is being shared across client and server side).
>
> Promise to add some documentation on the wiki about this ASAP.
>
> -----Original Message-----
> From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
> Sent: Monday, August 09, 2010 2:22 PM
> To: <hive-dev@hadoop.apache.org>
> Subject: Hive local mode on by default for 0.6.0
>
> I already caugh someone on IRC who was very surprised by the local
> mode in hive trunk. Is local mode on by default?
>
> Do you think the release 0.6.0 should have this on by default? There
> have been a few issues like HIVE-1520, and it seems like letting this
> out in the wild without actively turning it on might find edge cases
> and complications.
>
> Regards,
> Edward
>

Another thing is some people launch jobs from machines without
multiple-disks and cores, sometimes even from the namenode. I think on
these machines local performance will be poor or even dangerous. e.g.
OOM crashes name node and corrupts FSimage.

Someone came to the IRC and wondered why jobs were not showing in the
job tracker. I personally feel like it would be better to turn this
off. People that know about it and want it on can set it to true.

It is a super cool feature though.

Mime
View raw message