Return-Path: X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 441B8114C9 for ; Wed, 13 Aug 2014 06:54:34 +0000 (UTC) Received: (qmail 20618 invoked by uid 500); 13 Aug 2014 06:54:34 -0000 Delivered-To: apmail-falcon-dev-archive@falcon.apache.org Received: (qmail 20579 invoked by uid 500); 13 Aug 2014 06:54:34 -0000 Mailing-List: contact dev-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list dev@falcon.incubator.apache.org Received: (qmail 20563 invoked by uid 99); 13 Aug 2014 06:54:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 06:54:34 +0000 X-ASF-Spam-Status: No, hits=-2000.7 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 13 Aug 2014 06:54:32 +0000 Received: (qmail 20442 invoked by uid 99); 13 Aug 2014 06:54:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 13 Aug 2014 06:54:12 +0000 Date: Wed, 13 Aug 2014 06:54:12 +0000 (UTC) From: "Satish Mittal (JIRA)" To: dev@falcon.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FALCON-596) Enhancement required in Hcat feed replication MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/FALCON-596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14095221#comment-14095221 ] Satish Mittal commented on FALCON-596: -------------------------------------- Also partitions.put() is called at 2 places in CatalogStorge. You might need to update there as well. > Enhancement required in Hcat feed replication > --------------------------------------------- > > Key: FALCON-596 > URL: https://issues.apache.org/jira/browse/FALCON-596 > Project: Falcon > Issue Type: Improvement > Components: replication > Reporter: Suhas Vasu > Assignee: Suhas Vasu > Priority: Minor > Attachments: FALCON-596.patch > > > Hcatalog always creates partitions in lower case, hence falcon should always look for partitions by converting them to lower case > Ex: > (source table) > CREATE EXTERNAL TABLE in_table (word STRING, cnt INT) PARTITIONED BY (Year STRING, Month STRING, Date STRING, Hour STRING, Minute STRING) STORED AS TEXTFILE LOCATION '/user/suhas/hcat-data/in/'; > (target table) > CREATE EXTERNAL TABLE replicated_in_table (word STRING, cnt INT) PARTITIONED BY (Year STRING, Month STRING, Date STRING, Hour STRING, Minute STRING) STORED AS TEXTFILE LOCATION '/user/suhas/hcat-data/replicated_in/'; > The partitions in Hcat will be: > year=2014/month=08/date=05/hour=00/minute=00 > year=2014/month=08/date=05/hour=00/minute=01 > year=2014/month=08/date=05/hour=00/minute=02 > year=2014/month=08/date=05/hour=00/minute=03 > year=2014/month=08/date=05/hour=00/minute=04 > In such case falcon replication is failing with: > {noformat} > org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Exception fetching partition:org.apache.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : Invalid partition-key specified: year > at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:58) > at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:39) > at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:36) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438) > at org.apache.hadoop.mapred.Child.main(Child.java:262) > Caused by: org.apache.falcon.FalconException: Exception fetching partition:org.apache.hcatalog.common.HCatException : 9001 : Exception occurred while processing HCat request : Invalid partition-key specified: year > at org.apache.falcon.catalog.HiveCatalogService.getPartition(HiveCatalogService.java:235) > at org.apache.falcon.latedata.LateDataHandler.getTablePartitionCreateTimeMetric(LateDataHandler.java:243) > at org.apache.falcon.latedata.LateDataHandler.computeStorageMetric(LateDataHandler.java:183) > at org.apache.falcon.latedata.LateDataHandler.computeMetrics(LateDataHandler.java:124) > at org.apache.falcon.latedata.LateDataHandler.run(LateDataHandler.java:101) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.falcon.latedata.LateDataHandler.main(LateDataHandler.java:60) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:55) > ... 15 more > {noformat} -- This message was sent by Atlassian JIRA (v6.2#6252)