Return-Path: X-Original-To: apmail-falcon-dev-archive@minotaur.apache.org Delivered-To: apmail-falcon-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 05E04186FB for ; Tue, 11 Aug 2015 08:29:49 +0000 (UTC) Received: (qmail 1279 invoked by uid 500); 11 Aug 2015 08:23:09 -0000 Delivered-To: apmail-falcon-dev-archive@falcon.apache.org Received: (qmail 1228 invoked by uid 500); 11 Aug 2015 08:23:09 -0000 Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.apache.org Delivered-To: mailing list dev@falcon.apache.org Received: (qmail 1217 invoked by uid 99); 11 Aug 2015 08:23:09 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2015 08:23:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 1E9451A9C73 for ; Tue, 11 Aug 2015 08:23:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.172 X-Spam-Level: ***** X-Spam-Status: No, score=5.172 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=3, KAM_LAZY_DOMAIN_SECURITY=1, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, T_RP_MATCHES_RCVD=-0.01] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 7o4obgptlFSE for ; Tue, 11 Aug 2015 08:23:06 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with SMTP id 19FA4428F6 for ; Tue, 11 Aug 2015 08:23:06 +0000 (UTC) Received: (qmail 1182 invoked by uid 99); 11 Aug 2015 08:23:05 -0000 Received: from reviews-vm.apache.org (HELO reviews.apache.org) (140.211.11.40) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Aug 2015 08:23:05 +0000 Received: from reviews.apache.org (localhost [127.0.0.1]) by reviews.apache.org (Postfix) with ESMTP id 559F61DB600; Tue, 11 Aug 2015 08:23:03 +0000 (UTC) Content-Type: multipart/alternative; boundary="===============5474026998819494588==" MIME-Version: 1.0 Subject: Re: Review Request 35833: FALCON-1188: Falcon support for Hive Replication From: "Ajay Yadava" To: "Srikanth Sundarrajan" Cc: "Venkat Ranganathan" , "Falcon" , "Ajay Yadava" Date: Tue, 11 Aug 2015 08:23:03 -0000 Message-ID: <20150811082303.30461.25054@reviews.apache.org> X-ReviewBoard-URL: https://reviews.apache.org/ Auto-Submitted: auto-generated Sender: "Ajay Yadava" X-ReviewGroup: Falcon X-Auto-Response-Suppress: DR, RN, OOF, AutoReply X-ReviewRequest-URL: https://reviews.apache.org/r/35833/ X-Sender: "Ajay Yadava" References: <20150811030818.1375.89200@reviews.apache.org> In-Reply-To: <20150811030818.1375.89200@reviews.apache.org> Reply-To: "Ajay Yadava" X-ReviewRequest-Repository: falcon-git --===============5474026998819494588== MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35833/#review94891 ----------------------------------------------------------- Ship it! Ship It! - Ajay Yadava On Aug. 11, 2015, 3:08 a.m., Venkat Ranganathan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/35833/ > ----------------------------------------------------------- > > (Updated Aug. 11, 2015, 3:08 a.m.) > > > Review request for Falcon and Srikanth Sundarrajan. > > > Bugs: FALCON-1188 > https://issues.apache.org/jira/browse/FALCON-1188 > > > Repository: falcon-git > > > Description > ------- > > Falcon support for HiveDR > > As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR > > It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr. > > This supports HiveDR in both secure and unsecure clusters. > > This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues > > This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks > > > Diffs > ----- > > Installation-steps.txt bb92a85 > addons/hivedr/README PRE-CREATION > addons/hivedr/pom.xml PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION > addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION > addons/hivedr/src/main/resources/log4j.xml PRE-CREATION > addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION > addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION > addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION > addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION > addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5 > addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489 > addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459 > addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION > addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml PRE-CREATION > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml PRE-CREATION > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties PRE-CREATION > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml PRE-CREATION > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml PRE-CREATION > addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties PRE-CREATION > client/pom.xml afa91c9 > client/src/main/java/org/apache/falcon/cli/FalconCLI.java 148f789 > client/src/main/java/org/apache/falcon/client/FalconClient.java 9649e10 > client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION > client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION > client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION > client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION > client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION > client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION > client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f > client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846 > client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd > client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION > docs/src/site/twiki/InstallationSteps.twiki 3dd034b > oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4 > pom.xml 34a5471 > replication/pom.xml 8c4d6b4 > src/main/assemblies/distributed-package.xml 1e48c94 > src/main/assemblies/standalone-package.xml 682d52f > test-tools/hadoop-webapp/pom.xml cadc7f5 > webapp/pom.xml 063d42c > webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59 > webapp/src/test/resources/hdfs-replication-template.xml PRE-CREATION > webapp/src/test/resources/hdfs-replication.properties PRE-CREATION > webapp/src/test/resources/process.properties 1eb282c > > Diff: https://reviews.apache.org/r/35833/diff/ > > > Testing > ------- > > HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently. > Please see the README on component version requirements. > > > Thanks, > > Venkat Ranganathan > > --===============5474026998819494588==--