Return-Path: X-Original-To: apmail-falcon-commits-archive@minotaur.apache.org Delivered-To: apmail-falcon-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6158B1183D for ; Fri, 12 Sep 2014 20:29:57 +0000 (UTC) Received: (qmail 53268 invoked by uid 500); 12 Sep 2014 20:29:57 -0000 Delivered-To: apmail-falcon-commits-archive@falcon.apache.org Received: (qmail 53228 invoked by uid 500); 12 Sep 2014 20:29:57 -0000 Mailing-List: contact commits-help@falcon.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.incubator.apache.org Delivered-To: mailing list commits@falcon.incubator.apache.org Received: (qmail 53219 invoked by uid 99); 12 Sep 2014 20:29:57 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Sep 2014 20:29:57 +0000 X-ASF-Spam-Status: No, hits=-2001.7 required=5.0 tests=ALL_TRUSTED,RP_MATCHES_RCVD X-Spam-Check-By: apache.org Received: from [140.211.11.3] (HELO mail.apache.org) (140.211.11.3) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 12 Sep 2014 20:29:32 +0000 Received: (qmail 51880 invoked by uid 99); 12 Sep 2014 20:29:29 -0000 Received: from tyr.zones.apache.org (HELO tyr.zones.apache.org) (140.211.11.114) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Sep 2014 20:29:29 +0000 Received: by tyr.zones.apache.org (Postfix, from userid 65534) id AAC3DA12201; Fri, 12 Sep 2014 20:29:29 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: venkatesh@apache.org To: commits@falcon.incubator.apache.org Date: Fri, 12 Sep 2014 20:29:30 -0000 Message-Id: In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [2/5] git commit: FALCON-636 Add a sample recipe for disaster recovery of hdfs dirs/files. Contributed by Sowmya Ramesh X-Virus-Checked: Checked by ClamAV on apache.org FALCON-636 Add a sample recipe for disaster recovery of hdfs dirs/files. Contributed by Sowmya Ramesh Project: http://git-wip-us.apache.org/repos/asf/incubator-falcon/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-falcon/commit/42a1b767 Tree: http://git-wip-us.apache.org/repos/asf/incubator-falcon/tree/42a1b767 Diff: http://git-wip-us.apache.org/repos/asf/incubator-falcon/diff/42a1b767 Branch: refs/heads/master Commit: 42a1b76778274eb1cc289b7e4225db63944a7572 Parents: c54c1ef Author: Venkatesh Seetharam Authored: Fri Sep 12 13:26:08 2014 -0700 Committer: Venkatesh Seetharam Committed: Fri Sep 12 13:26:08 2014 -0700 ---------------------------------------------------------------------- CHANGES.txt | 3 + addons/recipes/hdfs-replication/pom.xml | 32 +++++++++ .../resources/hdfs-replication-template.xml | 45 +++++++++++++ .../resources/hdfs-replication-workflow.xml | 69 ++++++++++++++++++++ .../main/resources/hdfs-replication.properties | 63 ++++++++++++++++++ 5 files changed, 212 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/42a1b767/CHANGES.txt ---------------------------------------------------------------------- diff --git a/CHANGES.txt b/CHANGES.txt index b85ca8b..5764c47 100755 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -5,6 +5,9 @@ Trunk (Unreleased) INCOMPATIBLE CHANGES NEW FEATURES + FALCON-636 Add a sample recipe for disaster recovery of hdfs dirs/files + (Sowmya Ramesh via Venkatesh Seetharam) + FALCON-635 Add recipe option in falcon client (Sowmya Ramesh via Venkatesh Seetharam) http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/42a1b767/addons/recipes/hdfs-replication/pom.xml ---------------------------------------------------------------------- diff --git a/addons/recipes/hdfs-replication/pom.xml b/addons/recipes/hdfs-replication/pom.xml new file mode 100644 index 0000000..24ce24a --- /dev/null +++ b/addons/recipes/hdfs-replication/pom.xml @@ -0,0 +1,32 @@ + + + + + + + 4.0.0 + org.apache.falcon.recipes + falcon-hdfs-replication-recipe + 0.6-incubating-SNAPSHOT + Apache Falcon Sample Hdfs Replicaiton Recipe + Apache Falcon Sample Hdfs Replication Recipe + jar + http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/42a1b767/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml ---------------------------------------------------------------------- diff --git a/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml new file mode 100644 index 0000000..67c82db --- /dev/null +++ b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml @@ -0,0 +1,45 @@ + + + + + + + + + + + + 1 + LAST_ONLY + ##process.frequency## + UTC + + + + + + + + + + + + + + http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/42a1b767/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml ---------------------------------------------------------------------- diff --git a/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml new file mode 100644 index 0000000..7043e42 --- /dev/null +++ b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml @@ -0,0 +1,69 @@ + + + + + + + ${jobTracker} + ${nameNode} + + + oozie.launcher.mapreduce.job.user.classpath.first + true + + + oozie.launcher.mapreduce.user.classpath.first + true + + + mapred.job.queue.name + ${queueName} + + + oozie.launcher.mapred.job.priority + ${jobPriority} + + + oozie.use.system.libpath + true + + + org.apache.falcon.replication.FeedReplicator + -Dmapred.job.queue.name=${queueName} + -Dmapred.job.priority=${jobPriority} + -maxMaps + ${maxMaps} + -sourcePaths + ${nameNode}${drSourceDir} + -targetPath + ${drTargetClusterFS}${drTargetDir} + -falconFeedStorageType + FILESYSTEM + ${wf:conf("falcon.libpath")}/hadoop-distcp.jar + + + + + + + Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}] + + + + http://git-wip-us.apache.org/repos/asf/incubator-falcon/blob/42a1b767/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties ---------------------------------------------------------------------- diff --git a/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties new file mode 100644 index 0000000..34d1843 --- /dev/null +++ b/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties @@ -0,0 +1,63 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +##### Recipe properties +falcon.recipe.name=hdfs-replication + + +##### Workflow properties + +falcon.recipe.workflow.name=hdfs-dr-workflow +# If artifacts are present on local FS, provide paths here +falcon.recipe.workflow.path=/recipes/hdfs-replication/hdfs-replication-workflow.xml +falcon.recipe.workflow.lib.path=/recipes/hdfs-replication/lib + + +##### Cluster properties + +# Change the src cluster name here +falcon.recipe.src.cluster.name=test +# Change the src cluster hdfs write end point here. This is mandatory. +falcon.recipe.src.cluster.hdfs.writeEndPoint=hdfs://sandbox.hortonworks.com:8020 +# Change the src cluster validity start time here +falcon.recipe.src.cluster.validity.start=2012-04-20T00:00Z +# Change the src cluster validity end time here +falcon.recipe.src.cluster.validity.end=2014-04-20T00:00Z + + +##### Scheduling properties + +# Change the process here. Valid frequency type are minutes, hours, days, months +falcon.recipe.process.frequency=minutes(60) + + +##### Custom Job properties + +# Specify property names and values for properties defined in recipe template +falcon.recipe.process.property2.name=drSourceDir +falcon.recipe.process.property2.value=/falcon/test/srcCluster/input +falcon.recipe.process.property3.name=drTargetClusterFS +falcon.recipe.process.property3.value=hdfs://sandbox.hortonworks.com:8020 +falcon.recipe.process.property4.name=drTargetDir +falcon.recipe.process.property4.value=/falcon/test/targetCluster/input +falcon.recipe.process.property5.name=drTargetCluster +falcon.recipe.process.property5.value=backupCluster +falcon.recipe.process.property6.name=maxMaps +falcon.recipe.process.property6.value=5 +falcon.recipe.process.property7.name=mapBandwidth +falcon.recipe.process.property7.value=102400