Mailing-List: contact pig-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pig-dev@hadoop.apache.org
Message-ID: <7780679.126101280784916848.JavaMail.jira@thor>
Date: Mon, 2 Aug 2010 17:35:16 -0400 (EDT)
From: "Olga Natkovich (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Subject: [jira] Commented: (PIG-1526) HiveColumnarLoader Partitioning
 Support
In-Reply-To: <635663.85291280484797139.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/PIG-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894719#action_12894719 ] 

Olga Natkovich commented on PIG-1526:
-------------------------------------

Is this ready to be committed?

> HiveColumnarLoader Partitioning Support
> ---------------------------------------
>
>                 Key: PIG-1526
>                 URL: https://issues.apache.org/jira/browse/PIG-1526
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: 0.8.0
>            Reporter: Gerrit Jansen van Vuuren
>            Assignee: Gerrit Jansen van Vuuren
>            Priority: Minor
>             Fix For: 0.8.0
>
>         Attachments: PIG-1526.patch
>
>
> I've made allot improvements on the HiveColumnarLoader:
> -> Added support for LoadMetadata and data path Partitioning 
> -> Improved and simplefied column loading
> Data Path Partitioning:
> Hive stores partitions as folders like to /mytable/partition1=[value]/partition2=[value]. That is the table mytable contains 2 partitions [partition1, partition2].
> The HiveColumnarLoader will scan the inputpath /mytable and add to the PigSchema the columns partition2 and partition2. 
> These columns can then be used in filtering. 
> For example: We've got year,month,day,hour partitions in our data uploads.
> So a table might look like mytable/year=2010/month=02/day=01.
> Loading with the HiveColumnarLoader allows our pig scripts do filter by date using the standard pig Filter operator.
> I've added 2 classes for this:
> -> PathPartitioner
> -> PathPartitionHelper
> These classes are not hive dependent and could be used by any other loader that wants to support partitioning and helps with implementing the LoadMetadata interface.
> For this reason I though it best to put it into the package org.apache.pig.piggybank.storage.partition.
> What would be nice is in the future have the PigStorage also use these 2 classes to provide automatic path partitioning support. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.