incubator-hcatalog-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashutosh Chauhan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HCATALOG-61) HCatalogLoader should allow specifying a partition spec as a constructor argument
Date Sat, 16 Jul 2011 00:00:03 GMT

    [ https://issues.apache.org/jira/browse/HCATALOG-61?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066317#comment-13066317
] 

Ashutosh Chauhan commented on HCATALOG-61:
------------------------------------------

Why not following:
{code}
A = LOAD '...' USING HCatLoader();
B = Filter a by t = 4 or t = 5 or t = 6;
{code}

it will achieve the same thing and is much more natural to read and understand.

> HCatalogLoader should allow specifying a partition spec as a constructor argument
> ---------------------------------------------------------------------------------
>
>                 Key: HCATALOG-61
>                 URL: https://issues.apache.org/jira/browse/HCATALOG-61
>             Project: HCatalog
>          Issue Type: Improvement
>            Reporter: Julien Le Dem
>
> In some cases we want to pass the list of partitions to read as a parameter to the Pig
Script.
> This enables incremental processing. Reading a few new partitions in the input and generating
a new corresponding partition in the output.
> something similar to the partspec in the HCatStorer: 
> {code}
> pig -p "INPUT_PARTITIONS=t=4,t=5,t=6" -p "OUTPUT_PARTITION=t=2" myscript.pig
> {code}
> myscript.pig
> {code}
> A = LOAD '...' USING HCatLoader('$INPUT_PARTITIONS');
> ...
> STORE Z INTO '...' USING HCatStorer('$OUTPUT_PARTITION');
> {code}
> The number of partitions could be variable from one run to the other.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message