falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-90) Propagate hive table through Process mapper
Date Mon, 30 Sep 2013 19:22:25 GMT

    [ https://issues.apache.org/jira/browse/FALCON-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782119#comment-13782119
] 

Srikanth Sundarrajan commented on FALCON-90:
--------------------------------------------

Should we add a "." between prefix and property name ? Also should we rename this method to
propogateCatalogProperties instead since Storage is common to both FS & catalog. Perhaps
we can keep pigFilter and other filter instead of filter for each action type. (

{code}
    private void propagateStorageProperties(Input input, CatalogStorage tableStorage,
                                            Map<String, String> props) {
        String prefix = input.getName();

        props.put(prefix + "StorageType", tableStorage.getType().name());
        props.put(prefix + "CatalogUrl", tableStorage.getCatalogUrl());
        props.put(prefix + "Database", tableStorage.getDatabase());
        props.put(prefix + "Table", tableStorage.getTable());

        props.put(prefix + "PigPartitionFilter", "${coord:dataInPartitionFilter('input', 'pig')}");
        props.put(prefix + "HivePartitionFilter", "${coord:dataInPartitionFilter('input',
'hive')}");
        props.put(prefix + "JavaPartitionFilter", "${coord:dataInPartitionFilter('input',
'java')}");
    }
{code}


LateRerunConsumer::detectLate assumes falconInputFeeds & falconInPaths are populated.
however they are not populated in case hcat tables are used for process. Can we ensure that
this code path is not executed if input is hcat till the late handling is wired up for hcat
tables ?

Should propagateStorageProperties() also prefix + "." + property names ?

> Propagate hive table through Process mapper
> -------------------------------------------
>
>                 Key: FALCON-90
>                 URL: https://issues.apache.org/jira/browse/FALCON-90
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.3
>            Reporter: Venkatesh Seetharam
>            Assignee: Venkatesh Seetharam
>         Attachments: FALCON-90.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message