hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HIVE-543) provide option to run hive in local mode
Date Fri, 11 Jun 2010 02:55:13 GMT

     [ https://issues.apache.org/jira/browse/HIVE-543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joydeep Sen Sarma updated HIVE-543:
-----------------------------------

    Attachment: hive-543.patch.1

a few fixes for better local mode execution:
- provide alternate log4j file configuration for capturing local mode execution log (and default
to hive log4j if none provided). this cleans up the goop on the cli but allows captuing execution
time logs in a separate location if desired
- bypass distributed cache for local mode submissions. saves on hdfs time
  - some cleanup on the set/get MapRedWork code path. it seems to have been messed up after
the parallel execution changes
- getMRScratchDir - now returns a local scratch dir when executing in local mode. so we don't
hit hdfs unnecessarily in local mode.
- fix to fileutils.makequalified because of the above. there was a subtle bug in this that
was causing file paths to get messed up when using local paths for interemediate data
- bypassed query plan serialization/deserialization except for test mode. from past experience
- xml serialization/deserialization is pretty expensive and makes no sense to subject every
query to it.


> provide option to run hive in local mode
> ----------------------------------------
>
>                 Key: HIVE-543
>                 URL: https://issues.apache.org/jira/browse/HIVE-543
>             Project: Hadoop Hive
>          Issue Type: Improvement
>            Reporter: Joydeep Sen Sarma
>            Assignee: Joydeep Sen Sarma
>         Attachments: hive-543.patch.1
>
>
> this is a little bit more than just mapred.job.tracker=local
> when run in this mode - multiple jobs are an issue since writing to same tmp directories
is an issue. the following options:
> hadoop.tmp.dir
> mapred.local.dir
> need to be randomized (perhaps based on queryid). 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message