Return-Path: X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 38C291004B for ; Mon, 30 Sep 2013 19:22:59 +0000 (UTC) Received: (qmail 71257 invoked by uid 500); 30 Sep 2013 19:22:54 -0000 Delivered-To: apmail-falcon-dev-archive@falcon.apache.org Received: (qmail 71200 invoked by uid 500); 30 Sep 2013 19:22:54 -0000 Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list dev@falcon.incubator.apache.org Received: (qmail 70816 invoked by uid 99); 30 Sep 2013 19:22:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Sep 2013 19:22:52 +0000 X-ASF-Spam-Status: No, hits=-2000.7 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Mon, 30 Sep 2013 19:22:47 +0000 Received: (qmail 69546 invoked by uid 99); 30 Sep 2013 19:22:25 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 30 Sep 2013 19:22:25 +0000 Date: Mon, 30 Sep 2013 19:22:25 +0000 (UTC) From: "Srikanth Sundarrajan (JIRA)" To: dev@falcon.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FALCON-90) Propagate hive table through Process mapper MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/FALCON-90?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13782119#comment-13782119 ] Srikanth Sundarrajan commented on FALCON-90: -------------------------------------------- Should we add a "." between prefix and property name ? Also should we rename this method to propogateCatalogProperties instead since Storage is common to both FS & catalog. Perhaps we can keep pigFilter and other filter instead of filter for each action type. ( {code} private void propagateStorageProperties(Input input, CatalogStorage tableStorage, Map props) { String prefix = input.getName(); props.put(prefix + "StorageType", tableStorage.getType().name()); props.put(prefix + "CatalogUrl", tableStorage.getCatalogUrl()); props.put(prefix + "Database", tableStorage.getDatabase()); props.put(prefix + "Table", tableStorage.getTable()); props.put(prefix + "PigPartitionFilter", "${coord:dataInPartitionFilter('input', 'pig')}"); props.put(prefix + "HivePartitionFilter", "${coord:dataInPartitionFilter('input', 'hive')}"); props.put(prefix + "JavaPartitionFilter", "${coord:dataInPartitionFilter('input', 'java')}"); } {code} LateRerunConsumer::detectLate assumes falconInputFeeds & falconInPaths are populated. however they are not populated in case hcat tables are used for process. Can we ensure that this code path is not executed if input is hcat till the late handling is wired up for hcat tables ? Should propagateStorageProperties() also prefix + "." + property names ? > Propagate hive table through Process mapper > ------------------------------------------- > > Key: FALCON-90 > URL: https://issues.apache.org/jira/browse/FALCON-90 > Project: Falcon > Issue Type: Sub-task > Affects Versions: 0.3 > Reporter: Venkatesh Seetharam > Assignee: Venkatesh Seetharam > Attachments: FALCON-90.patch > > -- This message was sent by Atlassian JIRA (v6.1#6144)