hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lefty Leverenz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-1408) add option to let hive automatically run in local mode based on tunable heuristics
Date Wed, 21 Oct 2015 06:45:27 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14966328#comment-14966328
] 

Lefty Leverenz commented on HIVE-1408:
--------------------------------------

HIVE-9575 describes a problem with it in release 0.14.0.

The configuration parameters *hive.exec.mode.local.auto* and *hive.exec.mode.local.auto.inputbytes.max*
still exist in Hive 1.0 and above (but note that the default for *hive.exec.mode.local.auto*
is false -- it was changed in release 0.7.0 by some other JIRA issue).

The *hive.exec.mode.local.auto.tasks.max* parameter was changed to *hive.exec.mode.local.auto.input.files.max*
in release 0.9.0 by HIVE-2651.

> add option to let hive automatically run in local mode based on tunable heuristics
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-1408
>                 URL: https://issues.apache.org/jira/browse/HIVE-1408
>             Project: Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Joydeep Sen Sarma
>            Assignee: Joydeep Sen Sarma
>             Fix For: 0.7.0
>
>         Attachments: 1408.1.patch, 1408.2.patch, 1408.2.q.out.patch, 1408.7.patch, hive-1408.6.patch
>
>
> as a followup to HIVE-543 - we should have a simple option (enabled by default) to let
hive run in local mode if possible.
> two levels of options are desirable:
> 1. hive.exec.mode.local.auto=true/false // control whether local mode is automatically
chosen
> 2. Options to control different heuristics, some naiive examples:
>      hive.exec.mode.local.auto.input.size.max=1G // don't choose local mode if data >
1G
>      hive.exec.mode.local.auto.script.enable=true/false // choose if local mode is enabled
for queries with user scripts
> this can be implemented as a pre/post execution hook. It makes sense to provide this
as a standard hook in the hive codebase since it's likely to improve response time for many
users (especially for test queries).
> the initial proposal is to choose this at a query level and not at per hive-task (ie.
hadoop job) level. per job-level requires more changes to compilation (to not pre-commit to
hdfs or local scratch directories at compile time).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message