incubator-hcatalog-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Le Dem (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HCATALOG-61) HCatalogLoader should allow specifying a partition spec as a constructor argument
Date Fri, 15 Jul 2011 23:32:00 GMT
HCatalogLoader should allow specifying a partition spec as a constructor argument
---------------------------------------------------------------------------------

                 Key: HCATALOG-61
                 URL: https://issues.apache.org/jira/browse/HCATALOG-61
             Project: HCatalog
          Issue Type: Improvement
            Reporter: Julien Le Dem


In some cases we want to pass the list of partitions to read as a parameter to the Pig Script.
This enables incremental processing. Reading a few new partitions in the input and generating
a new corresponding partition in the output.
something similar to the partspec in the HCatStorer: 
{code}
pig -p "INPUT_PARTITIONS=t=4,t=5,t=6" -p "OUTPUT_PARTITION=t=2" myscript.pig
{code}
myscript.pig
{code}
A = LOAD '...' USING HCatLoader('$INPUT_PARTITIONS');
...
STORE Z INTO '...' USING HCatStorer('$OUTPUT_PARTITION');
{code}

The number of partitions could be variable from one run to the other.




--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message