falcon-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From b...@apache.org
Subject [08/49] falcon git commit: FALCON-2006 Update documentation on site for 0.10 release
Date Mon, 08 Aug 2016 23:15:42 GMT
http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/HiveMirroring.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/HiveMirroring.twiki b/trunk/general/src/site/twiki/HiveMirroring.twiki
new file mode 100644
index 0000000..e28502a
--- /dev/null
+++ b/trunk/general/src/site/twiki/HiveMirroring.twiki
@@ -0,0 +1,63 @@
+---+Hive Mirroring
+
+---++Overview
+Falcon provides feature to replicate Hive metadata and data events from source cluster to destination cluster. This is supported for both secure and unsecure cluster through Falcon extensions.
+
+---++Prerequisites
+Following is the prerequisites to use Hive Mirrroring
+
+   * *Hive 1.2.0+*
+   * *Oozie 4.2.0+*
+
+*Note:* Set following properties in hive-site.xml for replicating the Hive events on source and destination Hive cluster:
+<verbatim>
+    <property>
+        <name>hive.metastore.event.listeners</name>
+        <value>org.apache.hive.hcatalog.listener.DbNotificationListener</value>
+        <description>event listeners that are notified of any metastore changes</description>
+    </property>
+
+    <property>
+        <name>hive.metastore.dml.events</name>
+        <value>true</value>
+    </property>
+</verbatim>
+
+---++ Use Case
+* Replicate data/metadata of Hive DB & table from source to target cluster
+
+---++ Limitations
+* Currently Hive doesn't support create database, roles, views, offline tables, direct HDFS writes without registering with metadata and Database/Table name mapping replication events. Hence Hive mirroring extension cannot be used to replicate above mentioned events between warehouses.
+
+---++ Usage
+---+++ Bootstrap
+   Perform initial bootstrap of Table and Database from source cluster to destination cluster
+   * *Database Bootstrap*
+     For bootstrapping DB replication, first destination DB should be created. This step is expected,
+     since DB replication definitions can be set up by users only on pre-existing DB’s. Second, Export all tables in
+     the source db and Import it in the destination db, as described in Table bootstrap.
+
+   * *Table Bootstrap*
+     For bootstrapping table replication, essentially after having turned on the !DbNotificationListener
+     on the source db, perform an Export of the table, distcp the Export over to the destination
+     warehouse and do an Import over there. Check the following [[https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport][Hive Export-Import]] for syntax details
+     and examples.
+     This will set up the destination table so that the events on the source cluster that modify the table
+     will then be replicated.
+
+---+++  Setup source and destination clusters
+   <verbatim>
+    $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.xml
+   </verbatim>
+
+---+++ Hive mirroring extension properties
+   Extension artifacts are expected to be installed on HDFS at the path specified by "extension.store.uri" in startup properties. hive-mirroring-properties.json file located at "<extension.store.uri>/hive-mirroring/META/hive-mirroring-properties.json" lists all the required and optional parameters/arguments for scheduling Hive mirroring job.
+
+---+++ Submit and schedule Hive mirroring extension
+
+   <verbatim>
+    $FALCON_HOME/bin/falcon extension -submitAndSchedule -extensionName hive-mirroring -file /process/definition.xml
+   </verbatim>
+
+   Please Refer to [[falconcli/FalconCLI][Falcon CLI]] and [[restapi/ResourceList][REST API]] for more details on usage of CLI and REST API's.
+

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/InstallationSteps.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/InstallationSteps.twiki b/trunk/general/src/site/twiki/InstallationSteps.twiki
index a5ee2cc..297d88e 100644
--- a/trunk/general/src/site/twiki/InstallationSteps.twiki
+++ b/trunk/general/src/site/twiki/InstallationSteps.twiki
@@ -20,21 +20,22 @@ $git clone https://git-wip-us.apache.org/repos/asf/falcon.git falcon
 ---+++Step 2 - Build Falcon
 
 <verbatim>
-$cd falcon
-$export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify" && mvn clean install
+$ cd falcon
+$ export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=256m -noverify"
+$ mvn clean install 
+
 </verbatim>
 It builds and installs the package into the local repository, for use as a dependency in other projects locally.
 
-[optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of Hadoop]
-
-*NOTE:* Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards
-[optionally -Doozie.version=<<oozie version>> can be appended to build with a specific version of Oozie. Oozie versions
->= 4 are supported]
-NOTE: Falcon builds with JDK 1.7/1.8 using -noverify option
-      To compile Falcon with Hive Replication, optionally "-P hadoop-2,hivedr" can be appended. For this Hive >= 1.2.0
-      and Oozie >= 4.2.0 should be available.
+[optionally -Dhadoop.version=<<hadoop.version>> can be appended to build for a specific version of hadoop]
 
+*Note 1:* Falcon drops support for Hadoop-1 and only supports Hadoop-2 from Falcon 0.6 onwards
+          Falcon build with JDK 1.7 using -noverify option
 
+*Note 2:* To compile Falcon with addon extensions, append additional profiles to build command using syntax -P<<profile1,profile2>>
+          For Hive Mirroring extension, use profile"hivedr". Hive >= 1.2.0 and Oozie >= 4.2.0 is required
+          For HDFS Snapshot mirroring extension, use profile "hdfs-snapshot-mirroring". Hadoop >= 2.7.0 is required
+          For ADF integration, use profile "adf"
 
 ---+++Step 3 - Package and Deploy Falcon
 
@@ -75,6 +76,8 @@ $src/bin/package.sh <<hadoop-version>> <<oozie-version>>
 >> ex. src/bin/package.sh 2.5.0 4.0.0
 >> Falcon package is available in <<falcon home>>/target/apache-falcon-<<version>>-bin.tar.gz
 >> Oozie package is available in <<falcon home>>/target/oozie-4.0.1-distro.tar.gz
+>> __IMPORTANT:  You need to download the je-5.0.73 version from http://download.oracle.com/otn/berkeley-db/je-5.0.73.zip and extract je-5.0.73 under the Falcon webapp directory or provision an HBase cluster for use as Falcon graphdb backend DB.
+    Depending on the Graphdb backend choice, update the startup.properties appropriately.__
 </verbatim>
 
 *NOTE:* If you have a separate Apache Oozie installation, you will need to follow some additional steps:

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/MigrationInstructions.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/MigrationInstructions.twiki b/trunk/general/src/site/twiki/MigrationInstructions.twiki
index 7c0e027..a11dbc4 100644
--- a/trunk/general/src/site/twiki/MigrationInstructions.twiki
+++ b/trunk/general/src/site/twiki/MigrationInstructions.twiki
@@ -1,15 +1,32 @@
 ---+ Migration Instructions
 
----++ Migrate from 0.5-incubating to 0.6-incubating
+---++ Migrate from 0.9 to 0.10
 
-This is a placeholder wiki for migration instructions from falcon 0.5-incubating to 0.6-incubating.
+FALCON-1333 (Instance Search feature) requires Falcon to use titan-berkeleyje version 0.5.4 to support indexing.
+Up until version 0.9 - Falcon used titan-berkeleyje-jre6 version 0.4.2. GraphDB created by version 0.4.2 cannot be
+read by version 0.5.4. The solution is to migrate the GraphDB to be compatible with Falcon 0.10 release. Please make
+sure that no falcon server is running while performing the migration.
 
----+++ Update Entities
+---+++ 1. Install Falcon 0.10
+Install Falcon 0.10 by following the [[InstallationSteps][Installation Steps]]. Do not start the falcon server yet.
+The tool to migrate graphDB is packaged with 0.10 Falcon server in falcon-common-0.10.jar.
 
----+++ Change cluster dir permissions
+---+++ 2. Export GraphDB to JSON file using Falcon 0.9
+Please run the following command to generate the JSON file.
 
----+++ Enable/Disable TLS
+<verbatim>
+ $FALCON_HOME/bin/graphdbutil.sh export <<java_home> <<hadoop_home>> <<falcon_0.9_home>> <<path_to_falcon-common-0.10.jar>> /jsonFile/dir/
+</verbatim>
 
----+++ Authorization
+This command will create /jsonFile/dir/instanceMetadata.json
 
+---+++ 3. Import GraphDB from JSON file using Falcon 0.10
+Please run the following command to import graphDB the JSON file. The location of graphDB will be based on property
+"*.falcon.graph.storage.directory" set in startup.properties file.
+
+<verbatim>
+  $FALCON_HOME/bin/graphdbutil.sh export <<java_home> <<hadoop_home>> <<falcon_0.10_home>> <<path_to_falcon-common-0.10.jar>> /jsonFile/dir/
+</verbatim>
+
+This command will import from /jsonFile/dir/instanceMetadata.json, now start the Falcon 0.10 server.
 

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/Operability.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/Operability.twiki b/trunk/general/src/site/twiki/Operability.twiki
index 05850c1..2bccb51 100644
--- a/trunk/general/src/site/twiki/Operability.twiki
+++ b/trunk/general/src/site/twiki/Operability.twiki
@@ -72,22 +72,137 @@ or send alerts according to their requirements.
 
 ---++ Notifications
 
-Falcon creates a JMS topic for every process/feed that is scheduled in Falcon.
-The implementation class and the broker url of the JMS engine are read from the dependent cluster's definition.
-Users may register consumers on the required topic to check the availability or status of feed instances.
-
-For a given process that is scheduled, the name of the topic is same as the process name.
-Falcon sends a Map message for every feed produced by the instance of a process to the JMS topic.
-The JMS !MapMessage sent to a topic has the following properties:
-entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL;
-
-For a given feed that is scheduled, the name of the topic is same as the feed name.
-Falcon sends a map message for every feed instance that is deleted/archived/replicated depending upon the retention policy set in the feed definition.
-The JMS !MapMessage sent to a topic has the following properties:
-entityName, feedNames, feedInstancePath, workflowId, runId, nominalTime, timeStamp, brokerUrl, brokerImplClass, entityType, operation, logFile, topicName, status, brokerTTL;
-
-The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping service.TTL (Time-to-live) for JMS message
-can be configured in the Falcon's startup.properties file.
+Falcon has two types of notifications - System and User notifications.
+
+---+++ System notifications
+The System notifications are internally generated and used by Falcon to monitor the Falcon orchestrated workflow jobs.
+By default, Falcon starts an ActiveMQ embedded JMS server on Falcon machine on port 61616 as a daemon. Alternatively,
+users can make Falcon to use an existing JMS server instead of starting an embedded instance by doing the
+following 2 steps:
+
+   * Setting the property broker.url in the startup.properties as below
+<verbatim>
+   *.broker.url=tcp://jms-server-host:61616
+</verbatim>
+   * Set the system property falcon.embeddedmq to false as below
+<verbatim>
+   <FALCON-INSTALL-DIR>/bin/falcon-start -Dfalcon.embeddedmq=false
+</verbatim>
+
+Falcon uses FALCON.ENTITY.TOPIC to publish system notifications. This topic and the Map Message fields are internal
+and could change between releases.
+
+---+++ User notifications
+
+Falcon, in addition to the FALCON.ENTITY.TOPIC, also creates a JMS topic for every process/feed that is scheduled in
+Falcon as part of User notification. To enable User notifications, the broker url and implementation class of the JMS
+engine need to be specified in the cluster definition associated with the feed/process. Users may register consumers
+on the required topic to check the availability or status of feed instances. The User notification JMS broker instance
+can be same as the System notification or different.
+
+The name of the JMS topic is same as the process/feed name. Falcon sends a map message for every feed instance that is
+created/deleted/replicated/imported/exported to the JMS topic. The JMS Map Message sent to a topic has the following
+fields:
+
+   1. cluster - name of the current cluster the feed/process is dependent on.
+   1. entityType - type of the entity (feed or process).
+   1. entityName - name of the entity.
+   1. nominalTime - instance time (or data date).
+   1. operation - operation like generate, delete, replicate, import, export.
+   1. feedNames - name of the feeds which are generated/replicated/deleted/imported/exported.
+   1. feedInstancePaths - comma separated feed instance paths.
+   1. workflowId - current workflow-id of the instance.
+   1. workflowUser - user who owns the feed instance (i.e partition).
+   1. runId - current run-id of the instance.
+   1. status - status of the user workflow instance.
+   1. timeStamp - current timestamp.
+   1. logDir - log dir where lineage can be recorded.
+
+The JMS messages are automatically purged after a certain period (default 3 days) by the Falcon JMS house-keeping
+service. TTL (Time-to-live) for JMS message can be configured in the Falcon's startup.properties file.
+
+The following example shows how to enable and read user notification by connecting to the JMS broker.
+
+First, specify the JMS broker url in the cluster definition XML as shown below.
+
+<verbatim>
+
+<?xml version="1.0"?>
+<!-- filename : primaryCluster.xml -->
+<cluster colo="USWestOregon" description="oregonHadoopCluster" name="primaryCluster" xmlns="uri:falcon:cluster:0.1">
+    <interfaces>
+        ...
+        ...
+        <interface type="messaging" endpoint="tcp://user-jms-broker-host:61616?daemon=true" version="5.1.6" />
+        ...
+    </interfaces>
+</cluster>
+
+</verbatim>
+
+Next, use a JMS consumer (example below in Java) to read the message from the topic with the name
+FALCON.<feed-or-process-name>
+
+<verbatim>
+import org.apache.activemq.ActiveMQConnectionFactory;
+import org.apache.activemq.command.ActiveMQMapMessage;
+import javax.jms.ConnectionFactory;
+import javax.jms.Connection;
+import javax.jms.MessageConsumer;
+import javax.jms.Topic;
+import javax.jms.Session;
+import javax.jms.TopicSession;
+
+public class FalconUserJMSClient {
+    public static void main(String[] args)throws Exception {
+        // Note: specify the JMS broker URL
+        String brokerUrl = "tcp://localhost:61616";
+
+        ConnectionFactory connectionFactory = new ActiveMQConnectionFactory(brokerUrl);
+        Connection connection = connectionFactory.createConnection();
+        connection.setClientID("Falcon User JMS Consumer");
+        TopicSession session = (TopicSession) connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
+        try {
+
+            // Note: the topic name for the feed will be FALCON.<feed-name>
+            Topic falconTopic = session.createTopic("FALCON.feed-sample");
+            MessageConsumer consumer = session.createConsumer(falconTopic);
+            connection.start();
+            while (true) {
+                ActiveMQMapMessage msg = (ActiveMQMapMessage) consumer.receive();
+                System.out.println("cluster             : " + msg.getString("cluster"));
+                System.out.println("entityType          : " + msg.getString("entityType"));
+                System.out.println("entityName          : " + msg.getString("entityName"));
+                System.out.println("nominalTime         : " + msg.getString("nominalTime"));
+                System.out.println("operation           : " + msg.getString("operation"));
+
+                System.out.println("feedNames           : " + msg.getString("feedNames"));
+                System.out.println("feedInstancePaths   : " + msg.getString("feedInstancePaths"));
+
+                System.out.println("workflowId          : " + msg.getString("workflowId"));
+                System.out.println("workflowUser        : " + msg.getString("workflowUser"));
+                System.out.println("runId               : " + msg.getString("runId"));
+                System.out.println("status              : " + msg.getString("status"));
+                System.out.println("timeStamp           : " + msg.getString("timeStamp"));
+                System.out.println("logDir              : " + msg.getString("logDir"));
+
+                System.out.println("brokerUrl           : " + msg.getString("brokerUrl"));
+                System.out.println("brokerImplClass     : " + msg.getString("brokerImplClass"));
+                System.out.println("logFile             : " + msg.getString("logFile"));
+                System.out.println("topicName           : " + msg.getString("topicName"));
+                System.out.println("brokerTTL           : " + msg.getString("brokerTTL"));
+            }
+        } finally {
+            if (session != null) {
+                session.close();
+            }
+            if (connection != null) {
+                connection.close();
+            }
+        }
+    }
+}
+</verbatim>
 
 
 ---++ Alerts
@@ -108,3 +223,8 @@ under ${user.dir}/logs/ directory.
 
 Users may also extend the Falcon Audit plugin to send audits to systems like Apache Argus, etc. by
 extending org.apache.falcon.plugin.AuditingPlugin interface.
+
+
+---++ Metrics Collection In Graphite
+
+Falcon has support to send metrics to graphite more details regarding this can be found on [[GraphiteMetricCollection][Graphite Metric Collection]]
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/Security.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/Security.twiki b/trunk/general/src/site/twiki/Security.twiki
index 8955bdc..b17650c 100644
--- a/trunk/general/src/site/twiki/Security.twiki
+++ b/trunk/general/src/site/twiki/Security.twiki
@@ -2,8 +2,30 @@
 
 ---++ Overview
 
-Apache Falcon enforces authentication and authorization which are detailed below. Falcon also
-provides transport level security ensuring data confidentiality and integrity.
+Apache Falcon provides the following security features:
+   * Support credential provider alias for passwords used in Falcon server.
+   * Support authentication to identify proper users.
+   * Support authorization to specify resource access permission for users or groups.
+   * Support SSL to provide transport level security for data confidentiality and integrity.
+
+
+---++ Credential Provider Alias for Passwords
+Server-side configuration properties (i.e. startup.properties) contain passwords and other sensitive information.
+In addition to specifying properties in plain text, we provide the user an option to use credential provider alias in the property file.
+
+Take SMTP password for example. The user can store the password in a
+[[http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/CommandsManual.html#credential][Hadoop credential provider]]
+with the alias name _SMTPPasswordAlias_. In startup.properties where SMTP password is needed, the user can refer to its
+alias name _SMTPPasswordAlias_ instead of providing the real password.
+
+The alias property to be resolved through Hadoop credential provider should have the format:
+_credential.provider.alias.for.[property-key]_. For example,
+_credential.provider.alias.for.falcon.email.smtp.password=SMTPPasswordAlias_ for SMTP password.
+Falcon server, during the start, will automatically retrieve the real password provided the alias name.
+
+The user can specify the provider path with the property key _credential.provider.path_,
+e.g. _credential.provider.path=jceks://file/tmp/test.jceks_.
+If not specified, Falcon will use the default Hadoop credential provider path in core-site.xml.
 
 
 ---++ Authentication (User Identity)

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/DefineExtension.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/DefineExtension.twiki b/trunk/general/src/site/twiki/falconcli/DefineExtension.twiki
new file mode 100644
index 0000000..c260911
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/DefineExtension.twiki
@@ -0,0 +1,8 @@
+---+++Definition
+
+[[CommonCLI][Common CLI Options]]
+
+Definition of an extension. Outputs a JSON document describing the extension invocation parameters.
+
+Usage:
+$FALCON_HOME/bin/falcon extension ­definition ­name <extension­name>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki b/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki
index f2b3080..cc07406 100644
--- a/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki
+++ b/trunk/general/src/site/twiki/falconcli/DeleteEntity.twiki
@@ -2,7 +2,7 @@
 
 [[CommonCLI][Common CLI Options]]
 
-Delete removes the submitted entity definition for the specified entity and put it into the archive.
+Delete removes the submitted entity definition for the specified entity and put it into the archive.Archive path is defined in startup.properties in variable "config.store.uri".
 
 Usage:
 $FALCON_HOME/bin/falcon entity  -type [cluster|datasource|feed|process] -name <<name>> -delete

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/DescribeExtension.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/DescribeExtension.twiki b/trunk/general/src/site/twiki/falconcli/DescribeExtension.twiki
new file mode 100644
index 0000000..9f9895e
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/DescribeExtension.twiki
@@ -0,0 +1,8 @@
+---+++Describe
+
+[[CommonCLI][Common CLI Options]]
+
+Description of an extension. Outputs the README of the specified extension.
+
+Usage:
+$FALCON_HOME/bin/falcon extension ­describe ­name <extension­name>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/EnumerateExtension.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/EnumerateExtension.twiki b/trunk/general/src/site/twiki/falconcli/EnumerateExtension.twiki
new file mode 100644
index 0000000..0b28630
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/EnumerateExtension.twiki
@@ -0,0 +1,8 @@
+---+++Enumerate
+
+[[CommonCLI][Common CLI Options]]
+
+List all the extensions supported. Returns total number of results and a list of server side extensions supported.
+
+Usage:
+$FALCON_HOME/bin/falcon extension ­enumerate
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionDelete.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionDelete.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionDelete.twiki
new file mode 100644
index 0000000..bb867b5
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionDelete.twiki
@@ -0,0 +1,15 @@
+---+++Delete
+
+[[CommonCLI][Common CLI Options]]
+
+Delete an extension job.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -delete -jobName <<extension-job-name>>
+
+Optional Args : -doAs <<user-name>>
+
+<a href="../restapi/ExtensionDelete.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -delete -jobName sales-monthly

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionInstances.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionInstances.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionInstances.twiki
new file mode 100644
index 0000000..765a5bf
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionInstances.twiki
@@ -0,0 +1,18 @@
+---+++Instances
+
+[[CommonCLI][Common CLI Options]]
+
+List instances of an extension job.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -instances -jobName <<extension-job-name>>
+
+Optional Args : -doAs <<user-name>>
+-start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'" -fields <<field1,field2>>
+-instanceStatus <<instance-status>> -orderBy <<order-by-field>> -sortOrder <<sort-order>>
+-offset <<offset-number>> -numResults <<number-of-results>>
+
+<a href="../restapi/ExtensionInstances.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -instances -jobName daily-health-bill -start 2012-04-01T00:00

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionList.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionList.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionList.twiki
new file mode 100644
index 0000000..8ddf573
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionList.twiki
@@ -0,0 +1,16 @@
+---+++List
+
+[[CommonCLI][Common CLI Options]]
+
+List jobs generated from an extension.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -list -extensionName <<extension-name>>
+
+Optional Args : -doAs <<user-name>>
+-sortOrder <<sortOrder>> -offset <<offset-number>> -numResults <<number-of-results>> -fields <<field1,field2>>
+
+<a href="../restapi/ExtensionList.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -list -extensionName hdfs-mirroring -offset 0 -numResults 10

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionResume.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionResume.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionResume.twiki
new file mode 100644
index 0000000..4a62d04
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionResume.twiki
@@ -0,0 +1,15 @@
+---+++Resume
+
+[[CommonCLI][Common CLI Options]]
+
+Resume an extension job.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -resume -jobName <<extension-job-name>>
+
+Optional Args : -doAs <<user-name>>
+
+<a href="../restapi/ExtensionResume.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -resume -jobName sales-monthly

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionSchedule.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionSchedule.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionSchedule.twiki
new file mode 100644
index 0000000..813d7b7
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionSchedule.twiki
@@ -0,0 +1,15 @@
+---+++Schedule
+
+[[CommonCLI][Common CLI Options]]
+
+Schedule an extension job.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -schedule -jobName <<extension-job-name>>
+
+Optional Args : -doAs <<user-name>>
+
+<a href="../restapi/ExtensionSchedule.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -schedule -jobName sales-monthly

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionSubmit.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionSubmit.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionSubmit.twiki
new file mode 100644
index 0000000..40a7b44
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionSubmit.twiki
@@ -0,0 +1,15 @@
+---+++Submit
+
+[[CommonCLI][Common CLI Options]]
+
+Submit an extension job.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -submit -extensionName <<extension-name>> -file <<path-to-file>>
+
+Optional Args : -doAs <<user-name>>
+
+<a href="../restapi/ExtensionSubmit.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -submit -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki
new file mode 100644
index 0000000..ea1c0e2
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionSubmitAndSchedule.twiki
@@ -0,0 +1,15 @@
+---+++SubmitAndSchedule
+
+[[CommonCLI][Common CLI Options]]
+
+Submit and schedule an extension job.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -submitAndSchedule -extensionName <<extension-name>> -file <<path-to-file>>
+
+Optional Args : -doAs <<user-name>>
+
+<a href="../restapi/ExtensionSubmitAndSchedule.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -submitAndSchedule -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionSuspend.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionSuspend.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionSuspend.twiki
new file mode 100644
index 0000000..dd6b69c
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionSuspend.twiki
@@ -0,0 +1,15 @@
+---+++Suspend
+
+[[CommonCLI][Common CLI Options]]
+
+Suspend an extension job.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -suspend -jobName <<extension-job-name>>
+
+Optional Args : -doAs <<user-name>>
+
+<a href="../restapi/ExtensionSuspend.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -suspend -jobName sales-monthly

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionUpdate.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionUpdate.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionUpdate.twiki
new file mode 100644
index 0000000..fa0985e
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionUpdate.twiki
@@ -0,0 +1,15 @@
+---+++Update
+
+[[CommonCLI][Common CLI Options]]
+
+Update an extension entity. Note that we require all the entity names to remain the same. Future work will allow different entity names.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -update -extensionName <<extension-name>> -file <<path-to-file>>
+
+Optional Args : -doAs <<user-name>>
+
+<a href="../restapi/ExtensionUpdate.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -update -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/ExtensionValidate.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/ExtensionValidate.twiki b/trunk/general/src/site/twiki/falconcli/ExtensionValidate.twiki
new file mode 100644
index 0000000..d788a20
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/ExtensionValidate.twiki
@@ -0,0 +1,15 @@
+---+++Validate
+
+[[CommonCLI][Common CLI Options]]
+
+Validate the input for extension job submission.
+
+Usage:
+$FALCON_HOME/bin/falcon extension -validate -extensionName <<extension-name>> -file <<path-to-file>>
+
+Optional Args : -doAs <<user-name>>
+
+<a href="../restapi/ExtensionValidate.html">Parameters and examples described here.</a>
+
+Example:
+$FALCON_HOME/bin/falcon extension -validate -extensionName hdfs-mirroring -file /data/hdfs-mirroring-para.txt

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki b/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki
index 0c0082f..9ae0646 100644
--- a/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki
+++ b/trunk/general/src/site/twiki/falconcli/FalconCLI.twiki
@@ -11,7 +11,8 @@ CLI options are classified into :
    * <a href="#Instance_Management_Commands">Instance Management Commands</a>
    * <a href="#Metadata_Commands">Metadata Commands</a>
    * <a href="#Admin_Commands">Admin commands</a>
-   * <a href="#Recipe_Commands">Recipe commands</a>
+   * <a href="#Extension_Artifacts_Commands">Extension artifacts commands</a>
+   * <a href="#Extension_Commands">Extension commands</a>
 
 
 
@@ -68,7 +69,8 @@ $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.x
 | [[ContinueInstance][Continue]]                 | Continue the failed workflow instances          |
 | [[RerunInstance][Rerun]]                       | Rerun instances of specified process            |
 | [[ResumeInstance][Resume]]                     | Resume instance of specified process from suspended state   |
-| [[StatusInstance][Status]]                     | Gets the status of entity                       |
+| [[SearchInstance][Search]]                     | Search instances                                |
+| [[StatusInstance][Status]]                     | Gets the status of instances                    |
 | [[ListInstance][List]]                         | Gets single or multiple instances               |
 | [[SummaryInstance][Summary]]                   | Gets consolidated status of the instances between the specified time period    |
 | [[RunningInstance][Running]]                   | Gets running instances of the mentioned process |
@@ -100,13 +102,29 @@ $FALCON_HOME/bin/falcon entity -submit -type cluster -file /cluster/definition.x
 |[[HelpAdmin][Help]]                             | Return help options                             |
 |[[VersionAdmin][Version]]                       | Return current falcon version                   |
 |[[StatusAdmin][Status]]                         | Return the status of falcon                     |
+|[[SafemodeAdmin][SetSafeMode]]                  | Set/unset Falcon Server in safemode             |
 
 -----------
 
----++Recipe Commands
+---++Extension Artifacts Management Commands
 
 | *Command*                                      | *Description*                                   |
-|[[SubmitRecipe][Submit]]                        | Submit the specified Recipe                     |
-
+|[[EnumerateExtension][Enumerate]]               | Return all the extensions supported             |
+|[[DescribeExtension][Describe]]                 | Return description of an extension              |
+|[[DefineExtension][Definition]]                 | Return the definition of an extension           |
 
+-----------
 
+---++Extension Job Management Commands
+
+| *Command*                                         | *Description*                                   |
+|[[ExtensionList][List]]                            | List jobs generated from an extension           |
+|[[ExtensionInstances][Instances]]                  | List instances of an extension job              |
+|[[ExtensionValidate][Validate]]                    | Validate an extension job                       |
+|[[ExtensionSubmit][Submit]]                        | Submit an extension job                         |
+|[[ExtensionUpdate][Update]]                        | Update an extension job                         |
+|[[ExtensionSubmitAndSchedule][SubmitAndSchedule]]  | Submit and schedule an extension job            |
+|[[ExtensionSchedule][Schedule]]                    | Schedule an extension job                       |
+|[[ExtensionSuspend][Suspend]]                      | Suspend an extension job                        |
+|[[ExtensionResume][Resume]]                        | Resume an extension job                         |
+|[[ExtensionDelete][Delete]]                        | Delete an extension job                         |

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/SafemodeAdmin.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/SafemodeAdmin.twiki b/trunk/general/src/site/twiki/falconcli/SafemodeAdmin.twiki
new file mode 100644
index 0000000..5eb26c0
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/SafemodeAdmin.twiki
@@ -0,0 +1,10 @@
+---+++SetSafeMode
+
+[[CommonCLI][Common CLI Options]]
+
+When -setsafemode is true, Falcon starts in safemode.
+When -setsafemode is false, falcon starts in normal mode.
+
+Usage:
+$FALCON_HOME/bin/falcon admin -setsafemode true/false
+

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/SearchInstance.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/SearchInstance.twiki b/trunk/general/src/site/twiki/falconcli/SearchInstance.twiki
new file mode 100644
index 0000000..35cac74
--- /dev/null
+++ b/trunk/general/src/site/twiki/falconcli/SearchInstance.twiki
@@ -0,0 +1,17 @@
+---+++Search
+
+[[CommonCLI][Common CLI Options]]
+
+Search option via CLI can be used to get instances that meet search criteria, e.g. nominal time, instance status, entity name subsequence, entity tag, etc.
+Instance name and status are listed in the output. More instance properties will be added to the output in the future (e.g. log url, running time).
+Example usage include: search of instances of all entities with the same tag; search of all current running instances.
+
+Usage:
+$FALCON_HOME/bin/falcon instance -search
+
+Optional Args : -type <<[feed|process|datasource],[feed|process|datasource]>>
+-nameseq <<namesubsequence>> -tagkeys <<tagkeyword1,tagkeyword2>>
+-instanceStatus <<status>> -start "yyyy-MM-dd'T'HH:mm'Z'" -end "yyyy-MM-dd'T'HH:mm'Z'"
+-orderBy <<field>> -offset 0 -numResults 10
+
+<a href="../restapi/InstanceSearch.html">Optional params and examples described here.</a>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki b/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki
index ae60559..146a60f 100644
--- a/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki
+++ b/trunk/general/src/site/twiki/falconcli/UpdateEntity.twiki
@@ -2,13 +2,15 @@
 
 [[CommonCLI][Common CLI Options]]
 
-Update operation allows an already submitted/scheduled entity to be updated. Cluster and datasource updates are
-currently not allowed.
+Update operation allows an already submitted/scheduled entity to be updated and put it into the archive.Archive path is defined in startup.properties in variable "config.store.uri". Datasource updates are currently not allowed.
 
 Usage:
-$FALCON_HOME/bin/falcon entity  -type [feed|process] -name <<name>> -update -file <<path_to_file>>
+$FALCON_HOME/bin/falcon entity  -type [cluster|feed|process] -name <<name>> -update -file <<path_to_file>>
 
 Optional Arg : -skipDryRun. When this argument is specified, Falcon skips oozie dryrun.
 
 Example:
 $FALCON_HOME/bin/falcon entity -type process -name hourly-reports-generator -update -file /process/definition.xml
+
+Note: When a cluster entity is updated, the dependent feed and process bundle+coordinators are updated in the
+workflow engine. Hence, only a falcon superuser who has ability to impersonate other users can update a cluster entity.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/AdminSafemode.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/AdminSafemode.twiki b/trunk/general/src/site/twiki/restapi/AdminSafemode.twiki
new file mode 100644
index 0000000..d7c9631
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/AdminSafemode.twiki
@@ -0,0 +1,25 @@
+---++  GET /api/admin/setSafeMode/:mode
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Set falcon server into/out of safemode.
+
+---++ Parameters
+   * :mode true/false.
+   * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system.
+
+---++ Results
+The value of FalconSever safemode.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/admin/setSafeMode/true
+</verbatim>
+---+++ Result
+<verbatim>
+true
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/AdminVersion.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/AdminVersion.twiki b/trunk/general/src/site/twiki/restapi/AdminVersion.twiki
index 7db2d8f..d7615a0 100644
--- a/trunk/general/src/site/twiki/restapi/AdminVersion.twiki
+++ b/trunk/general/src/site/twiki/restapi/AdminVersion.twiki
@@ -29,6 +29,14 @@ GET http://localhost:15000/api/admin/version?doAs=joe
         {
             "key":"Mode",
             "value":"embedded"
+        },
+        {
+            "key":"authentication",
+            "value":"simple"
+        },
+        {
+            "key":"safemode",
+            "value":"false"
         }
     ]
 }

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/EntityList.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/EntityList.twiki b/trunk/general/src/site/twiki/restapi/EntityList.twiki
index 2c2a734..a439dc7 100644
--- a/trunk/general/src/site/twiki/restapi/EntityList.twiki
+++ b/trunk/general/src/site/twiki/restapi/EntityList.twiki
@@ -1,4 +1,4 @@
----++  GET /api/entities/list/:entity-type?fields=:fields
+---++  GET /api/entities/list/{:entity-type}
    * <a href="#Description">Description</a>
    * <a href="#Parameters">Parameters</a>
    * <a href="#Results">Results</a>
@@ -8,7 +8,7 @@
 Get list of the entities.
 
 ---++ Parameters
-   * :entity-type Comma-separated entity types. Can be empty. Valid entity types are cluster, feed or process.
+   * :entity-type <optional param> Comma-separated entity types. Valid entity types are cluster, feed or process.
    * fields <optional param> Fields of entity that the user wants to view, separated by commas.
       * Valid options are STATUS, TAGS, PIPELINES, CLUSTERS.
    * nameseq <optional param> Subsequence of entity name. Not case sensitive.
@@ -38,20 +38,28 @@ Total number of results and a list of entities.
 ---++ Examples
 ---+++ Rest Call
 <verbatim>
-GET http://localhost:15000/api/entities/list/feed
+GET http://localhost:15000/api/entities/list
 </verbatim>
 ---+++ Result
 <verbatim>
 {
-    "totalResults":"2”,
+    "totalResults":"4”,
     "entity": [
         {
-            "name": "SampleOutput",
-            "type": "feed"
+            "name"  : "SampleCluster1",
+            "type"  : "cluster"
+        }
+        {
+            "name"  : "SampleOutput",
+            "type"  : "feed"
         },
         {
-            "name": "SampleInput",
-            "type": "feed"
+            "name"  : "SampleInput",
+            "type"  : "feed"
+        }
+        {
+            "name"  : "SampleProcess1",
+            "type"  : "process"
         }
     ]
 }
@@ -59,28 +67,20 @@ GET http://localhost:15000/api/entities/list/feed
 
 ---+++ Rest Call
 <verbatim>
-GET http://localhost:15000/api/entities/list
+GET http://localhost:15000/api/entities/list/feed
 </verbatim>
 ---+++ Result
 <verbatim>
 {
-    "totalResults":"4”,
+    "totalResults":"2”,
     "entity": [
         {
-            "name"  : "SampleCluster1",
-            "type"  : "cluster"
-        }
-        {
-            "name"  : "SampleOutput",
-            "type"  : "feed"
+            "name": "SampleOutput",
+            "type": "feed"
         },
         {
-            "name"  : "SampleInput",
-            "type"  : "feed"
-        }
-        {
-            "name"  : "SampleProcess1",
-            "type"  : "process"
+            "name": "SampleInput",
+            "type": "feed"
         }
     ]
 }

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki b/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki
index 46b01fc..cbf33db 100644
--- a/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki
+++ b/trunk/general/src/site/twiki/restapi/EntityUpdate.twiki
@@ -8,8 +8,8 @@
 Updates the submitted entity.
 
 ---++ Parameters
-   * :entity-type can be feed or process.
-   * :entity-name is name of the feed or process.
+   * :entity-type can be cluster, feed or process.
+   * :entity-name is name of the cluster, feed or process.
    * skipDryRun : Optional query param, Falcon skips oozie dryrun when value is set to true.
    * doAs <optional query param> allows the current user to impersonate the user passed in doAs when interacting with the Falcon system.
 

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionDefinition.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionDefinition.twiki b/trunk/general/src/site/twiki/restapi/ExtensionDefinition.twiki
new file mode 100644
index 0000000..66f6674
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionDefinition.twiki
@@ -0,0 +1,160 @@
+---++  GET api/extension/definition/:extension­name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Get definition of the extension.
+
+---++ Parameters
+   * :extension­name Name of the extension.
+
+---++ Results
+Outputs a JSON document describing the extension invocation parameters.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/extension/definition/hdfs­mirroring
+</verbatim>
+---+++ Result
+<verbatim>
+{
+    "shortDescription": "This extension implements replicating arbitrary directories on HDFS from one Hadoop cluster to another Hadoop cluster. This piggy backs on replication solution in Falcon which uses the DistCp tool.",
+    "properties":[
+        {
+            "propertyName":"jobName",
+            "required":true,
+            "description":"Unique job name",
+            "example":"hdfs-monthly-sales-dr"
+        },
+        {
+            "propertyName":"jobClusterName",
+            "required":true,
+            "description":"Cluster where job should run",
+            "example":"backupCluster"
+        },
+        {
+            "propertyName":"jobValidityStart",
+            "required":true,
+            "description":"Job validity start time",
+            "example":"2016-03-03T00:00Z"
+        },
+        {
+            "propertyName":"jobValidityEnd",
+            "required":true,
+            "description":"Job validity end time",
+            "example":"2018-03-13T00:00Z"
+        },
+        {
+            "propertyName":"jobFrequency",
+            "required":true,
+            "description":"job frequency. Valid frequency types are minutes, hours, days, months",
+            "example":"months(1)"
+        },
+        {
+            "propertyName":"jobTimezone",
+            "required":false,
+            "description":"Time zone for the job",
+            "example":"GMT"
+        },
+        {
+            "propertyName":"jobTags",
+            "required":false,
+            "description":"list of comma separated tags. Key Value Pairs, separated by comma",
+            "example":"consumer=consumer@xyz.com, owner=producer@xyz.com, _department_type=forecasting"
+        },
+        {
+            "propertyName":"jobRetryPolicy",
+            "required":false,
+            "description":"Job retry policy",
+            "example":"periodic"
+        },
+        {
+            "propertyName":"jobRetryDelay",
+            "required":false,
+            "description":"Job retry delay",
+            "example":"minutes(30)"
+        },
+        {
+            "propertyName":"jobRetryAttempts",
+            "required":false,
+            "description":"Job retry attempts",
+            "example":"3"
+        },
+        {
+            "propertyName":"jobRetryOnTimeout",
+            "required":false,
+            "description":"Job retry on timeout",
+            "example":"true"
+        },
+        {
+            "propertyName":"jobAclOwner",
+            "required":false,
+            "description":"ACL owner",
+            "example":"ambari-qa"
+        },
+        {
+            "propertyName":"jobAclGroup",
+            "required":false,
+            "description":"ACL group",
+            "example":"users"
+        },
+        {
+            "propertyName":"jobAclPermission",
+            "required":false,
+            "description":"ACL permission",
+            "example":"0x755"
+        },
+        {
+            "propertyName":"sourceDir",
+            "required":true,
+            "description":"Multiple hdfs comma separated source directories",
+            "example":"/user/ambari-qa/primaryCluster/dr/input1, /user/ambari-qa/primaryCluster/dr/input2"
+        },
+        {
+            "propertyName":"sourceCluster",
+            "required":true,
+            "description":"Source cluster for hdfs mirroring",
+            "example":"primaryCluster"
+        },
+        {
+            "propertyName":"targetDir",
+            "required":true,
+            "description":"Target hdfs directory",
+            "example":"/user/ambari-qa/backupCluster/dr"
+        },
+        {
+            "propertyName":"targetCluster",
+            "required":true,
+            "description":"Target cluster for hdfs mirroring",
+            "example":"backupCluster"
+        },
+        {
+            "propertyName":"distcpMaxMaps",
+            "required":false,
+            "description":"Maximum number of mappers for DistCP",
+            "example":"1"
+        },
+        {
+            "propertyName":"distcpMapBandwidth",
+            "required":false,
+            "description":"Bandwidth in MB for each mapper in DistCP",
+            "example":"100"
+        },
+        {
+            "propertyName":"jobNotificationType",
+            "required":false,
+            "description":"Email Notification for Falcon instance completion",
+            "example":"email"
+        },
+        {
+            "propertyName":"jobNotificationReceivers",
+            "required":false,
+            "description":"Comma separated email Id's",
+            "example":"user1@gmail.com, user2@gmail.com"
+        }
+    ]
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionDelete.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionDelete.twiki b/trunk/general/src/site/twiki/restapi/ExtensionDelete.twiki
new file mode 100644
index 0000000..2bdd23c
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionDelete.twiki
@@ -0,0 +1,29 @@
+---++  POST api/extension/delete/:job-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Delete an extension job.
+
+---++ Parameters
+   * :job-name Name of the extension job.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Result of the delete operation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/extensions/delete/sales-monthly
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3",
+  "message": "Extension job sales-monthly deleted successfully",
+  "status": "SUCCEEDED"
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionDescription.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionDescription.twiki b/trunk/general/src/site/twiki/restapi/ExtensionDescription.twiki
new file mode 100644
index 0000000..5900fbb
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionDescription.twiki
@@ -0,0 +1,24 @@
+---++  GET api/extension/describe/:extension­name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Description of an extension.
+
+---++ Parameters
+   * :extension­name Name of the extension.
+
+---++ Results
+Outputs the README of the specified extension.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/extension/describe/hdfs­mirroring
+</verbatim>
+---+++ Result
+<verbatim>
+<README file of the specified extension>
+</verbatim>
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionEnumeration.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionEnumeration.twiki b/trunk/general/src/site/twiki/restapi/ExtensionEnumeration.twiki
new file mode 100644
index 0000000..abd94c8
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionEnumeration.twiki
@@ -0,0 +1,38 @@
+---++  GET api/extension/enumerate
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Get list of the supported extensions.
+
+---++ Parameters
+None
+
+---++ Results
+Total number of results and a list of extension server extensions supported.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/extension/enumerate
+</verbatim>
+---+++ Result
+<verbatim>
+{
+    "totalResults":"2”,
+    “extensions”: [
+        {
+            “name”: “Hdfs­mirroring”
+            “type”: “Trusted/Provided extension”
+            “description”: “This extension implements replicating arbitrary directories on HDFS from one Hadoop cluster to another Hadoop cluster.”
+        },
+        {
+            “name”: “Hive­mirroring”
+            “type”: “Trusted/Provided extension”
+            “description”: “This extension implements replicating hive metadata and data from one Hadoop cluster to another Hadoop cluster.”
+        }
+    ]
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionInstances.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionInstances.twiki b/trunk/general/src/site/twiki/restapi/ExtensionInstances.twiki
new file mode 100644
index 0000000..a11a97a
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionInstances.twiki
@@ -0,0 +1,87 @@
+---++  GET api/extension/instances/:job-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+List instances of an extension job.
+
+---++ Parameters
+   * :job-name Name of the extension job.
+   * start <optional> Show instances from this nominal time. Format: yyyy-MM-dd'T'HH:mm'Z'.
+   * end <optional> Show instances up to this nominal time. Format: yyyy-MM-dd'T'HH:mm'Z'.
+   * instanceStatus <optional> Valid options are RUNNING, SUCCEEDED, FAILED, WAITING, SUSPENDED, KILLED, TIMEOUT.
+   * fields <optional> Entity output fields separated by commas. Valid options: STATUS, TAGS, PIPELINES, CLUSTERS.
+   * orderBy <optional> Column by which results should be ordered. Sorted by descending order. Valid options are nominalTime (default) and status.
+   * sortOrder <optional> Valid options are “asc” and “desc”.
+   * offset <optional> Show results from the offset. Used for pagination. Default is 0.
+   * numResults <optional> Number of instances per entity to show. Default value is 10.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+A list of entities of the job, each followed by a set of instances.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/extensions/instances/daily-health-bill?start=2012-04-01T00:00
+</verbatim>
+---+++ Result
+<verbatim>
+{
+    "numEntities": "2",
+    "entitiesSummary": {
+        "entitySummary": [
+            {
+                "entityProfile": {
+                    "type": "FEED",
+                    "name": "SampleUSHealthBill"
+                },
+                "instances": [
+                    {
+                        "instance": "2012-04-02T00:00Z",
+                        "status": "RUNNING",
+                        "cluster": "SampleCluster1",
+                        "sourceCluster": "primaryCluster",
+                        "details": "hftp://sandbox.hortonworks.com:50070/apps/falcon/adf-demo/pig-input-2012-04-02"
+                    },
+                    {
+                        "instance": "2012-04-01T00:00Z",
+                        "status": "SUCCEEDED",
+                        "cluster": "SampleCluster1",
+                        "sourceCluster": "primaryCluster",
+                        "details": "hftp://sandbox.hortonworks.com:50070/apps/falcon/adf-demo/pig-input-2012-04-01"
+                    }
+                ]
+            },
+            {
+                "entityProfile": {
+                    "type": "PROCESS",
+                    "name": "SampleBillPay"
+                },
+                "instances": [
+                    {
+                        "instance": "2012-04-02T00:00Z",
+                        "status": "RUNNING",
+                        "logFile": "http://sandbox.hortonworks.com:11000/oozie?job=0000085-160322203213815-oozie-oozi-W",
+                        "cluster": "primaryCluster",
+                        "startTime": "2016-04-04T22:37:46Z",
+                        "details": "",
+                        "actions": {
+                            "action": "dr-replication",
+                            "status": "RUNNING",
+                            "logFile": "http://sandbox.hortonworks.com:8088/proxy/application_1458678764964_0052/"
+                        }
+                    },
+                    {
+                        "instance": "2012-04-01T00:00Z",
+                        "status": "SKIPPED",
+                        "cluster": "primaryCluster"
+                    }
+                ]
+            }
+        ]
+    }
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionList.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionList.twiki b/trunk/general/src/site/twiki/restapi/ExtensionList.twiki
new file mode 100644
index 0000000..dcd4d8d
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionList.twiki
@@ -0,0 +1,71 @@
+---++  GET api/extension/list/:extension-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+List jobs generated from an extension.
+
+---++ Parameters
+   * :extension-name Name of the extension.
+   * sortOrder <optional> Sort order by job name. Valid options: "asc" (default) and "desc".
+   * offset <optional> Show results from the offset. Default is 0.
+   * numResults <optional> Number of results to show per request. Default is 10.
+   * fields <optional> Output fields separated by commas. Valid options: STATUS, TAGS, PIPELINES, CLUSTERS.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Total number of results and a list of jobs generated from the extension, followed by the associated entities.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/extensions/list/billCollection?fields=status,clusters,tags
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "numJobs":"2",
+  "jobs":{
+    "job": [
+      {
+        "jobName": "daily-health-bill",
+        "jobEntities: {
+          "totalResults": "2",
+          "entity": [
+            {
+              "type":"FEED",
+              "name":"SampleUSHealthBill",
+              "status":"SUBMITTED”,
+              "tags":{"tag":["related=ushealthcare","department=billingDepartment","_falcon_extension_name=billCoollection","_falcon_extension_job=daily-health-bill"]},
+              "clusters": {"cluster":["SampleCluster1","primaryCluster”]}
+            },
+            {
+              "type":"PROCESS”,
+              "name":"SampleBillPay”,
+              "status":"RUNNING”,
+              "tags":{"tag":["related=healthcare","department=billingDepartment","_falcon_extension_name=billCoollection","_falcon_extension_job=daily-health-bill"]},
+              "clusters":{"cluster":"primaryCluster”}
+            }
+          ]
+        }
+      },
+      {
+        "jobName": "fsa-bill",
+        "jobEntities": {
+          "totalResults": "1",
+          "entity":
+              {
+                "type":"PROCESS”,
+                "name":"FSAPay”,
+                "status":"RUNNING”,
+                "tags”:{"tag":["related=healthcare","department=billingDepartment","_falcon_extension_name=billCollection","_falcon_extension_job=fsa-bill"]},
+                "clusters":{"cluster":"primaryCluster”}
+              }
+        }
+      }
+    ]
+  }
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionResume.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionResume.twiki b/trunk/general/src/site/twiki/restapi/ExtensionResume.twiki
new file mode 100644
index 0000000..70cbbff
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionResume.twiki
@@ -0,0 +1,29 @@
+---++  POST api/extension/resume/:job-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Resume an extension job.
+
+---++ Parameters
+   * :job-name Name of the extension job.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Result of the resume operation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/extensions/resume/sales-monthly
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3",
+  "message": "Extension job sales-monthly resumed successfully",
+  "status": "SUCCEEDED"
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionSchedule.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionSchedule.twiki b/trunk/general/src/site/twiki/restapi/ExtensionSchedule.twiki
new file mode 100644
index 0000000..c40ef6e
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionSchedule.twiki
@@ -0,0 +1,29 @@
+---++  POST api/extension/schedule/:job-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Schedule an extension job.
+
+---++ Parameters
+   * :job-name Name of the extension job.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Result of the schedule operation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/extensions/schedule/sales-monthly
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3",
+  "message": "Extension job sales-monthly scheduled successfully",
+  "status": "SUCCEEDED"
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionSubmit.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionSubmit.twiki b/trunk/general/src/site/twiki/restapi/ExtensionSubmit.twiki
new file mode 100644
index 0000000..6b4df32
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionSubmit.twiki
@@ -0,0 +1,38 @@
+---++  POST api/extension/submit/:extension-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Submit an extension job.
+
+---++ Parameters
+   * :extension-name Name of the extension.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Result of submission.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/extensions/submit/hdfs-mirroring
+jobName=sales-monthly
+jobClustername=primaryCluster
+jobClusterValidityStart=2015-03-13T00:00Z
+jobClusterValidityEnd=2016-12-30T00:00Z
+jobFrequency=minutes(5)
+sourceDir=/user/hrt_qa/dr/test/primaryCluster/input
+sourceCluster=primaryCluster
+targetDir=/user/hrt_qa/dr/test/backupCluster/input
+targetCluster=backupCluster
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3",
+  "message": "Extension job submitted successfully",
+  "status": "SUCCEEDED"
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionSubmitAndSchedule.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionSubmitAndSchedule.twiki b/trunk/general/src/site/twiki/restapi/ExtensionSubmitAndSchedule.twiki
new file mode 100644
index 0000000..8a148ce
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionSubmitAndSchedule.twiki
@@ -0,0 +1,38 @@
+---++  POST api/extension/submitAndSchedule/:extension-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Submit and schedule an extension job.
+
+---++ Parameters
+   * :extension-name Name of the extension.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Result of the submit and schedule operation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/extensions/submitAndSchedule/hdfs-mirroring
+jobName=sales-monthly
+jobClustername=primaryCluster
+jobClusterValidityStart=2015-03-13T00:00Z
+jobClusterValidityEnd=2016-12-30T00:00Z
+jobFrequency=minutes(5)
+sourceDir=/user/hrt_qa/dr/test/primaryCluster/input
+sourceCluster=primaryCluster
+targetDir=/user/hrt_qa/dr/test/backupCluster/input
+targetCluster=backupCluster
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3",
+  "message": "Extension job submitted and scheduled successfully",
+  "status": "SUCCEEDED"
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionSuspend.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionSuspend.twiki b/trunk/general/src/site/twiki/restapi/ExtensionSuspend.twiki
new file mode 100644
index 0000000..cc3478f
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionSuspend.twiki
@@ -0,0 +1,29 @@
+---++  POST api/extension/suspend/:job-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Suspend an extension job.
+
+---++ Parameters
+   * :job-name Name of the extension job.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Result of the suspend operation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/extensions/suspend/sales-monthly
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3",
+  "message": "Extension job sales-monthly suspended successfully",
+  "status": "SUCCEEDED"
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionUpdate.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionUpdate.twiki b/trunk/general/src/site/twiki/restapi/ExtensionUpdate.twiki
new file mode 100644
index 0000000..1ac626a
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionUpdate.twiki
@@ -0,0 +1,38 @@
+---++  POST api/extension/update/:extension-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Update an extension entity. Note that we require all the entity names to remain the same. Future work will allow different entity names.
+
+---++ Parameters
+   * :extension-name Name of the extension.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Result of update.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/extensions/update/hdfs-mirroring
+jobName=sales-monthly
+jobClustername=primaryCluster
+jobClusterValidityStart=2015-03-13T00:00Z
+jobClusterValidityEnd=2016-12-30T00:00Z
+jobFrequency=minutes(5)
+sourceDir=/user/hrt_qa/dr/test/primaryCluster/input
+sourceCluster=primaryCluster
+targetDir=/user/hrt_qa/dr/test/backupCluster/input
+targetCluster=backupCluster
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3",
+  "message": "Updated successfully",
+  "status": "SUCCEEDED"
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ExtensionValidate.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ExtensionValidate.twiki b/trunk/general/src/site/twiki/restapi/ExtensionValidate.twiki
new file mode 100644
index 0000000..c27f2d7
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/ExtensionValidate.twiki
@@ -0,0 +1,38 @@
+---++  POST api/extension/validate/:extension-name
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Validate the input for extension job submission.
+
+---++ Parameters
+   * :extension-name Name of the extension.
+   * doAs <optional> Impersonate the user.
+
+---++ Results
+Result of validation.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+POST http://localhost:15000/api/extensions/validate/hdfs-mirroring
+jobName=sales-monthly
+jobClustername=primaryCluster
+jobClusterValidityStart=2015-03-13T00:00Z
+jobClusterValidityEnd=2016-12-30T00:00Z
+jobFrequency=minutes(5)
+sourceDir=/user/hrt_qa/dr/test/primaryCluster/input
+sourceCluster=primaryCluster
+targetDir=/user/hrt_qa/dr/test/backupCluster/input
+targetCluster=backupCluster
+</verbatim>
+---+++ Result
+<verbatim>
+{
+  "requestId": "e5cc8230-f356-4566-9b65-536abdff8aa3",
+  "message": "Validated successfully",
+  "status": "SUCCEEDED"
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/InstanceSearch.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/InstanceSearch.twiki b/trunk/general/src/site/twiki/restapi/InstanceSearch.twiki
new file mode 100644
index 0000000..4744240
--- /dev/null
+++ b/trunk/general/src/site/twiki/restapi/InstanceSearch.twiki
@@ -0,0 +1,97 @@
+---++  GET /api/instance/search
+   * <a href="#Description">Description</a>
+   * <a href="#Parameters">Parameters</a>
+   * <a href="#Results">Results</a>
+   * <a href="#Examples">Examples</a>
+
+---++ Description
+Get the list of instances that meet search criteria.
+
+---++ Parameters
+   * :type <optional param> Comma-separated entity types. Valid entity types are feed, process or datasource.
+   * nameseq <optional param> Subsequence of entity name. Not case sensitive.
+      * The entity name needs to contain all the characters in the subsequence in the same order.
+      * Example 1: "sample1" will match the entity named "SampleFeed1-2".
+      * Example 2: "mhs" will match the entity named "New-My-Hourly-Summary".
+   * tagkeys <optional param> Keywords in tags, separated by comma. Not case sensitive.
+      * The returned entities will have tags that match all the tag keywords.
+   * instanceStatus <optional param> Valid options are RUNNING, SUCCEEDED, FAILED, WAITING, SUSPENDED, KILLED, TIMEOUT.
+   * start <optional param> Show instances from this nominal time. Format: yyyy-MM-dd'T'HH:mm'Z'.
+      * By default, it is set to (end - (10 * entityFrequency)).
+   * end <optional param> Show instances up to this nominal time. Format: yyyy-MM-dd'T'HH:mm'Z'.
+      * Default is set to now.
+   * orderBy <optional param> Field by which results should be ordered. Sorted by descending order.
+      * If not specified, instances of the same entity will be listed together in the result.
+      * Current valid option: status. Coming soon: sorting by name, nominal time, running time.
+   * offset <optional param> Show results from the offset, used for pagination. Defaults to 0.
+   * numResults <optional param> Number of results to show per request, used for pagination.
+      * Only integers > 0 are valid. Default is 10.
+   * doAs <optional query param> allows the current user to impersonate the user passed in doAs.
+
+---++ Results
+A list of instances.
+
+---++ Examples
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/instance/search?nameseq=usbill&start=2012-04-03T00:00Z
+</verbatim>
+---+++ Result
+<verbatim>
+{
+    "instances": [
+        {
+	 “name":"SampleUSHealthBill”,
+	 “type":"FEED”,
+	 “tags”: {"tag":["related=ushealthcare","department=billingDepartment"]},
+            “nominalTime": "2012-04-03T07:00Z"
+            "cluster": "primary-cluster",
+            "status": "SUCCEEDED",
+            "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W",
+            "startTime": "2013-10-21T14:39:56-07:00",
+            "endTime": "2013-10-21T14:40:26-07:00",
+        }
+    ],
+    "requestId": "default\/e15bb378-d09f-4911-9df2-5334a45153d2\n",
+    "message": "Instances Search Results",
+    "status": "SUCCEEDED”
+}
+</verbatim>
+
+---+++ Rest Call
+<verbatim>
+GET http://localhost:15000/api/instance/search?nameseq=samplebill&tagkeys=billing,healthcare&start=2012-04-03T00:00Z&instanceStatus=succeeded&orderBy=nominalTime&offset=2&numResults=2
+</verbatim>
+---+++ Result
+<verbatim>
+ {
+    "instances": [
+        {
+	 "name":"SampleHealthBill",
+	 "type":"PROCESS",
+	 "tags”: {"tag":["related=healthcare","department=billingDepartment"]},
+            “nominalTime": "2012-04-03T08:00Z"
+            "cluster": "primary-cluster",
+            "status": "SUCCEEDED",
+            "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933397-oozie-rgau-W",
+            "startTime": "2013-10-21T14:41:56-07:00",
+            "endTime": "2013-10-21T14:42:26-07:00",
+        },
+        {
+	 "name":"SampleUSHealthBill",
+	 "type":"FEED”,
+	 "tags": {"tag":["related=ushealthcare","department=billingDepartment"]},
+            “nominalTime": "2012-04-03T07:00Z"
+            "cluster": "primary-cluster",
+            "status": "SUCCEEDED",
+            "logFile": "http:\/\/localhost:11000\/oozie?job=0000070-131021115933395-oozie-rgau-W",
+            "startTime": "2013-10-21T14:39:56-07:00",
+            "endTime": "2013-10-21T14:40:26-07:00",
+        },
+    ],
+
+    "requestId": "default\/e15bb378-d09f-4911-9df2-5334a45153d2\n",
+    "message": "Instances Search Results",
+    "status": "SUCCEEDED”
+}
+</verbatim>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/general/src/site/twiki/restapi/ResourceList.twiki
----------------------------------------------------------------------
diff --git a/trunk/general/src/site/twiki/restapi/ResourceList.twiki b/trunk/general/src/site/twiki/restapi/ResourceList.twiki
index 34c2c6f..5a4c253 100644
--- a/trunk/general/src/site/twiki/restapi/ResourceList.twiki
+++ b/trunk/general/src/site/twiki/restapi/ResourceList.twiki
@@ -6,6 +6,8 @@
    * <a href="#REST_Call_on_Admin_Resource">REST Call on Admin Resource</a>
    * <a href="#REST_Call_on_Lineage_Graph">REST Call on Lineage Graph Resource</a>
    * <a href="#REST_Call_on_Metadata_Resource">REST Call on Metadata Resource</a>
+   * <a href="#REST_Call_on_Extension_Artifact">REST Call on Extension artifact</a>
+   * <a href="#REST_Call_on_Extension_Job Management">REST Call on Extension Job Management</a>
 
 ---++ Authentication
 
@@ -37,6 +39,7 @@ The current version of the rest api's documentation is also hosted on the Falcon
 | GET         | [[AdminStack][api/admin/stack]]                | Get stack of the server                     |
 | GET         | [[AdminVersion][api/admin/version]]            | Get version of the server                   |
 | GET         | [[AdminConfig][api/admin/config/:config-type]] | Get configuration information of the server |
+| GET         | [[AdminSafemode][api/admin/setSafeMode/:mode]] | Set safemode to true/false in falcon server |
 
 ---++ REST Call on Entity Resource
 
@@ -51,7 +54,7 @@ The current version of the rest api's documentation is also hosted on the Falcon
 | DELETE      | [[EntityDelete][api/entities/delete/:entity-type/:entity-name]]             | Delete the entity                  |
 | GET         | [[EntityStatus][api/entities/status/:entity-type/:entity-name]]             | Get the status of the entity       |
 | GET         | [[EntityDefinition][api/entities/definition/:entity-type/:entity-name]]     | Get the definition of the entity   |
-| GET         | [[EntityList][api/entities/list/:entity-type]]                              | Get the list of entities           |
+| GET         | [[EntityList][api/entities/list/{:entity-type}]]                            | Get the list of entities           |
 | GET         | [[EntitySummary][api/entities/summary/:entity-type/:cluster]]               | Get instance summary of all entities |
 | GET         | [[EntityDependencies][api/entities/dependencies/:entity-type/:entity-name]] | Get the dependencies of the entity |
 | GET         | [[FeedSLA][api/entities/sla-alert/:entity-type]]                            | Get pending feed instances which missed sla |
@@ -88,6 +91,28 @@ The current version of the rest api's documentation is also hosted on the Falcon
 
 ---++ REST Call on Metadata Discovery Resource
 
-| *Call Type* | *Resource*                                                                                     | *Description*                                                                 |
-| GET         | [[MetadataList][api/metadata/discovery/:dimension-type/list]]                                  | list of dimensions  |
-| GET         | [MetadataRelations][api/metadata/discovery/:dimension-type/:dimension-name/relations]]         | Return all relations of a dimension |
+| *Call Type* | *Resource*                                                                                     | *Description*                       |
+| GET         | [[MetadataList][api/metadata/discovery/:dimension-type/list]]                                  | list of dimensions                  |
+| GET         | [[MetadataRelations][api/metadata/discovery/:dimension-type/:dimension-name/relations]]        | Return all relations of a dimension |
+
+---++ REST Call on Extension Artifact
+
+| *Call Type* | *Resource*                                                        | *Description*                                                          |
+| GET         | [[ExtensionEnumeration][api/extension/enumerate]]                 | List all the extensions supported                                      |
+| GET         | [[ExtensionDescription][api/extension/describe/:extension­name]]  | Return the README of the specified extension                           |
+| GET         | [[ExtensionDefinition][api/extension/definition/:extension­name]] | Return a JSON document describing the extension invocation parameters  |
+
+
+---++ REST Call on Extension Job Management
+
+| *Call Type* | *Resource*                                                                       | *Description*                                                          |
+| GET         | [[ExtensionList][api/extension/list/:extension­name]]                            | List jobs generated from an extension                                  |
+| GET         | [[ExtensionInstances][api/extension/instances/:job-name]]                        | List instances of an extension job                                     |
+| POST        | [[ExtensionValidate][api/extension/validate/:extension­name]]                    | Validate an extension job                                              |
+| POST        | [[ExtensionSubmit][api/extension/submit/:extension­name]]                        | Submit an extension job                                                |
+| POST        | [[ExtensionUpdate][api/extension/update/:extension­name]]                        | Update an extension job                                                |
+| POST        | [[ExtensionSubmitAndSchedule][api/extension/submitAndSchedule/:extension­name]]  | Submit and schedule an extension job                                   |
+| POST        | [[ExtensionSchedule][api/extension/schedule/:job-name]]                          | Schedule an extension job                                              |
+| POST        | [[ExtensionSuspend][api/extension/suspend/:job-name]]                            | Suspend an extension job                                               |
+| POST        | [[ExtensionResume][api/extension/resume/:job-name]]                              | Resume an extension job                                                |
+| POST        | [[ExtensionDelete][api/extension/delete/:job-name]]                              | Delete an extension job                                                |
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/pom.xml
----------------------------------------------------------------------
diff --git a/trunk/pom.xml b/trunk/pom.xml
index a3e2557..b60eb84 100644
--- a/trunk/pom.xml
+++ b/trunk/pom.xml
@@ -21,7 +21,7 @@
     <modelVersion>4.0.0</modelVersion>
     <groupId>org.apache.falcon</groupId>
     <artifactId>falcon-website</artifactId>
-    <version>0.9-SNAPSHOT</version>
+    <version>0.11-SNAPSHOT</version>
     <packaging>pom</packaging>
 
     <name>Apache Falcon</name>
@@ -254,7 +254,7 @@
             <email>sowmya_kr@apache.org</email>
             <timezone>-08:00</timezone>
             <roles>
-                <role>committer</role>
+                <role>PPMC</role>
             </roles>
             <organization>Hortonworks</organization>
         </developer>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/releases/0.10/pom.xml
----------------------------------------------------------------------
diff --git a/trunk/releases/0.10/pom.xml b/trunk/releases/0.10/pom.xml
new file mode 100644
index 0000000..e9e6a21
--- /dev/null
+++ b/trunk/releases/0.10/pom.xml
@@ -0,0 +1,66 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+  
+       http://www.apache.org/licenses/LICENSE-2.0
+  
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>org.apache.falcon</groupId>
+        <artifactId>falcon-website-releases</artifactId>
+        <version>0.2</version>
+    </parent>
+    <artifactId>falcon-website-0.10</artifactId>
+    <version>0.10</version>
+    <packaging>pom</packaging>
+
+    <name>Apache Falcon - Documentation v0.10</name>
+
+    <build>
+        <plugins>
+            <plugin>
+                <groupId>org.apache.maven.plugins</groupId>
+                <artifactId>maven-site-plugin</artifactId>
+                <version>3.3</version>
+                <dependencies>
+                    <dependency>
+                        <groupId>org.apache.maven.doxia</groupId>
+                        <artifactId>doxia-module-twiki</artifactId>
+                        <version>1.3</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.maven.wagon</groupId>
+                        <artifactId>wagon-ssh-external</artifactId>
+                        <version>2.6</version>
+                    </dependency>
+                </dependencies>
+                <executions>
+                    <execution>
+                        <goals>
+                            <goal>site</goal>
+                        </goals>
+                        <phase>prepare-package</phase>
+                    </execution>
+                </executions>
+                <configuration>
+                    <outputDirectory>../../../content/0.10</outputDirectory>
+                </configuration>
+            </plugin>
+        </plugins>
+    </build>
+
+</project>

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/releases/0.10/src/site/resources/Architecture.png
----------------------------------------------------------------------
diff --git a/trunk/releases/0.10/src/site/resources/Architecture.png b/trunk/releases/0.10/src/site/resources/Architecture.png
new file mode 100644
index 0000000..0378b49
Binary files /dev/null and b/trunk/releases/0.10/src/site/resources/Architecture.png differ

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/releases/0.10/src/site/resources/EntityDependency.png
----------------------------------------------------------------------
diff --git a/trunk/releases/0.10/src/site/resources/EntityDependency.png b/trunk/releases/0.10/src/site/resources/EntityDependency.png
new file mode 100644
index 0000000..9f11870
Binary files /dev/null and b/trunk/releases/0.10/src/site/resources/EntityDependency.png differ

http://git-wip-us.apache.org/repos/asf/falcon/blob/4612c3f7/trunk/releases/0.10/src/site/resources/FeedSchedule.png
----------------------------------------------------------------------
diff --git a/trunk/releases/0.10/src/site/resources/FeedSchedule.png b/trunk/releases/0.10/src/site/resources/FeedSchedule.png
new file mode 100644
index 0000000..105c6b1
Binary files /dev/null and b/trunk/releases/0.10/src/site/resources/FeedSchedule.png differ


Mime
View raw message