hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <>
Subject [jira] Updated: (HIVE-467) Scratch data location should be on different filesystems for different types of intermediate data
Date Wed, 13 May 2009 16:33:45 GMT


Joydeep Sen Sarma updated HIVE-467:

    Attachment: hive-467.patch.2

- takes care of the issues mentioned here - consolidates all temp file logic into one place
with different policies for local/MR/external storage
a) DDL/Explain operations only require local storage - tests for the same included.
b) insert operations write to tmp storage on file system of target. ie. if the destination
is an external table on s3 - we will write to tmp storage on s3 and then move to target. unfortunately
- can't think of a way to test this since don't have two functioning file systems
c) all else get MR intermediate storage.

fixed a problem with external partitions where insert overwrites into an external partition
were not using the external location (but the default location). test for this is included.

review appreciated - i would like to publish a recipe for working against s3 storage and can't
really do without this.

> Scratch data location should be on different filesystems for different types of intermediate
> -------------------------------------------------------------------------------------------------
>                 Key: HIVE-467
>                 URL:
>             Project: Hadoop Hive
>          Issue Type: Bug
>          Components: Query Processor
>         Environment: S3/EC2
>            Reporter: Joydeep Sen Sarma
>            Assignee: Joydeep Sen Sarma
>         Attachments: hive-467.patch.1, hive-467.patch.2
> Currently Hive uses the same scratch directory/path for all sorts of temporary and intermediate
data. This is problematic:
> 1. Temporary location for writing out DDL output should just be temp file on local file
system. This divorces the dependence of metadata and browsing operations on a functioning
hadoop cluster.
> 2. Temporary location of intermediate map-reduce data should be the default file system
(which is typically the hdfs instance on the compute cluster)
> 3. Temporary location for data that needs to be 'moved' into tables should be on the
same file system as the table's location (table's location may not be same as hdfs instance
of processing cluster).
> ie. - local storage, map-reduce intermediate storage and table storage should be distinguished.
Without this distinction - using hive on environments like S3/EC2 causes problems. In such
an environment - i would like to be able to:
> - do metadata operations without a provisioned hadoop cluster (using data stored in S3
and metastore on local disk)
> - attach to a provisioned hadoop cluster and run queries
> - store data back in tables that are created over s3 file system

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message