hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerrit Jansen van Vuuren (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-1526) HiveColumnarLoader Partitioning Support
Date Fri, 30 Jul 2010 10:13:17 GMT
HiveColumnarLoader Partitioning Support

                 Key: PIG-1526
                 URL: https://issues.apache.org/jira/browse/PIG-1526
             Project: Pig
          Issue Type: Improvement
    Affects Versions: 0.8.0
            Reporter: Gerrit Jansen van Vuuren
            Assignee: Gerrit Jansen van Vuuren
             Fix For: 0.8.0

I've made allot improvements on the HiveColumnarLoader:
-> Added support for LoadMetadata and data path Partitioning 
-> Improved and simplefied column loading

Data Path Partitioning:

Hive stores partitions as folders like to /mytable/partition1=[value]/partition2=[value].
That is the table mytable contains 2 partitions [partition1, partition2].
The HiveColumnarLoader will scan the inputpath /mytable and add to the PigSchema the columns
partition2 and partition2. 
These columns can then be used in filtering. 
For example: We've got year,month,day,hour partitions in our data uploads.
So a table might look like mytable/year=2010/month=02/day=01.
Loading with the HiveColumnarLoader allows our pig scripts do filter by date using the standard
pig Filter operator.

I've added 2 classes for this:
-> PathPartitioner
-> PathPartitionHelper

These classes are not hive dependent and could be used by any other loader that wants to support
partitioning and helps with implementing the LoadMetadata interface.
For this reason I though it best to put it into the package org.apache.pig.piggybank.storage.partition.
What would be nice is in the future have the PigStorage also use these 2 classes to provide
automatic path partitioning support. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message