falcon-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sowmya...@apache.org
Subject falcon git commit: FALCON-1585 Documentation for HDFS and Hive DR. Contributed by Peeyush Bishnoi
Date Fri, 06 Nov 2015 19:48:22 GMT
Repository: falcon
Updated Branches:
  refs/heads/0.8 9d201294f -> 517718bfb


FALCON-1585 Documentation for HDFS and Hive DR. Contributed by Peeyush Bishnoi


Project: http://git-wip-us.apache.org/repos/asf/falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/falcon/commit/517718bf
Tree: http://git-wip-us.apache.org/repos/asf/falcon/tree/517718bf
Diff: http://git-wip-us.apache.org/repos/asf/falcon/diff/517718bf

Branch: refs/heads/0.8
Commit: 517718bfbdd30b09785de303b7b8b9a8f35dae29
Parents: 9d20129
Author: Sowmya Ramesh <sramesh@hortonworks.com>
Authored: Fri Nov 6 11:48:19 2015 -0800
Committer: Sowmya Ramesh <sramesh@hortonworks.com>
Committed: Fri Nov 6 11:48:19 2015 -0800

----------------------------------------------------------------------
 CHANGES.txt                       |  2 ++
 docs/src/site/twiki/HDFSDR.twiki  | 21 +++++++++++
 docs/src/site/twiki/HiveDR.twiki  | 64 ++++++++++++++++++++++++++++++++++
 docs/src/site/twiki/Recipes.twiki |  4 +++
 4 files changed, 91 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/CHANGES.txt
----------------------------------------------------------------------
diff --git a/CHANGES.txt b/CHANGES.txt
index 9eec6ea..7bf823d 100755
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -27,6 +27,8 @@ Release version: 0.8
     FALCON-1027 Falcon proxy user support(Sowmya Ramesh)
 
   IMPROVEMENTS
+    FALCON-1585 Documentation for HDFS and Hive DR(Peeyush Bishnoi via Sowmya Ramesh)
+
     FALCON-1582 Documentation for globally disabling retries (Pallavi Rao)
  
     FALCON-1574 Document behavior of lifecycle retention stage frequency behavior(Sowmya
Ramesh via Ajay Yadava).

http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/HDFSDR.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/HDFSDR.twiki b/docs/src/site/twiki/HDFSDR.twiki
new file mode 100644
index 0000000..4a42c4c
--- /dev/null
+++ b/docs/src/site/twiki/HDFSDR.twiki
@@ -0,0 +1,21 @@
+---+ HDFS DR Recipe
+---++ Overview
+Falcon supports HDFS DR recipe to replicate data from source cluster to destination cluster.
+
+---++ Usage
+---+++ Setup cluster definition.
+   <verbatim>
+    $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml
+   </verbatim>
+
+---+++ Update recipes properties
+   Update recipe properties file in addons/recipes/hdfs-replication with required attributes
for replicating
+   data from source cluster to destination cluster.
+
+---+++ Submit HDFS DR recipe
+   <verbatim>
+    $FALCON_HOME/bin/falcon recipe -name hdfs-replication -operation HDFS_REPLICATION
+   </verbatim>
+
+Recipe templates for HDFS DR is available in addons/recipes/hdfs-replication and copy it
to
+recipe path (*falcon.recipe.path=<recipe directory path>*) by specifying in client.properties.

http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/HiveDR.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/HiveDR.twiki b/docs/src/site/twiki/HiveDR.twiki
new file mode 100644
index 0000000..ca039ce
--- /dev/null
+++ b/docs/src/site/twiki/HiveDR.twiki
@@ -0,0 +1,64 @@
+---+Hive Disaster Recovery
+
+
+---++Overview
+Falcon provides feature to replicate Hive metadata and data events from source cluster
+to destination cluster. This is supported for secure and unsecure cluster through Falcon
Recipes.
+
+
+---++Prerequisites
+Following is the prerequisites to use Hive DR
+
+   * *Hive 1.2.0+*
+   * *Oozie 4.2.0+*
+
+*Note:* Set following properties in hive-site.xml for replicating the Hive events on source
and destination Hive cluster:
+<verbatim>
+    <property>
+        <name>hive.metastore.event.listeners</name>
+        <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
+        <description>event listeners that are notified of any metastore changes</description>
+    </property>
+
+    <property>
+        <name>hive.metastore.dml.events</name>
+        <value>true</value>
+    </property>
+</verbatim>
+
+---++ Usage
+---+++ Bootstrap
+   Perform initial bootstrap of Table and Database from source cluster to destination cluster
+   * *Database Bootstrap*
+     For bootstrapping DB replication, first destination DB should be created. This step
is expected,
+     since DB replication definitions can be set up by users only on pre-existing DB’s.
Second, Export all tables in
+     the source db and Import it in the destination db, as described in Table bootstrap.
+
+   * *Table Bootstrap*
+     For bootstrapping table replication, essentially after having turned on the !DbNotificationListener
+     on the source db, perform an Export of the table, distcp the Export over to the destination
+     warehouse and do an Import over there. Check the following [[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport][Hive
Export-Import]] for syntax details
+     and examples.
+     This will set up the destination table so that the events on the source cluster that
modify the table
+     will then be replicated.
+
+---+++ Setup cluster definition
+   <verbatim>
+    $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml
+   </verbatim>
+
+---+++ Update recipes properties
+   Update recipe properties file in addons/recipes/hive-disaster-recovery with required attributes
for replicating
+   Hive data and metadata from source cluster to destination cluster.
+
+---+++ Submit Hive DR recipe
+   <verbatim>
+    $FALCON_HOME/bin/falcon recipe -name hive-disaster-recovery -operation HIVE_DISASTER_RECOVERY
+   </verbatim>
+
+
+Recipe templates for Hive DR is available in addons/recipes/hive-disaster-recovery and copy
it to
+recipe path (*falcon.recipe.path=<recipe directory path>*) by specifying in client.properties.
+
+*Note:* If kerberos security is enabled on cluster, use the secure templates for Hive DR
from
+        addons/recipes/hive-disaster-recovery

http://git-wip-us.apache.org/repos/asf/falcon/blob/517718bf/docs/src/site/twiki/Recipes.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Recipes.twiki b/docs/src/site/twiki/Recipes.twiki
index ea29d94..792bf1b 100644
--- a/docs/src/site/twiki/Recipes.twiki
+++ b/docs/src/site/twiki/Recipes.twiki
@@ -76,6 +76,10 @@ GraphDB for display on UI.
 
    * Sample recipes are published in addons/recipes
 
+---++ Types of recipes
+   * [[HDFSDR][HDFS Recipe]]
+   * [[HiveDR][HiveDR Recipe]]
+
 ---++ Packaging
 
    * There is no packaging for recipes at this time but will be added soon.


Mime
View raw message