Return-Path: X-Original-To: apmail-falcon-commits-archive@minotaur.apache.org Delivered-To: apmail-falcon-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 332BB187ED for ; Mon, 15 Feb 2016 05:37:05 +0000 (UTC) Received: (qmail 59504 invoked by uid 500); 15 Feb 2016 05:37:05 -0000 Delivered-To: apmail-falcon-commits-archive@falcon.apache.org Received: (qmail 59468 invoked by uid 500); 15 Feb 2016 05:37:05 -0000 Mailing-List: contact commits-help@falcon.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@falcon.apache.org Delivered-To: mailing list commits@falcon.apache.org Received: (qmail 59459 invoked by uid 99); 15 Feb 2016 05:37:05 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 15 Feb 2016 05:37:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 8A43BC107C for ; Mon, 15 Feb 2016 05:37:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.794 X-Spam-Level: * X-Spam-Status: No, score=1.794 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-0.006] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id hRcRTpaGs_WX for ; Mon, 15 Feb 2016 05:37:01 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 9710F5FAF2 for ; Mon, 15 Feb 2016 05:37:00 +0000 (UTC) Received: from svn01-us-west.apache.org (svn.apache.org [10.41.0.6]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 0AF35E00B0 for ; Mon, 15 Feb 2016 05:37:00 +0000 (UTC) Received: from svn01-us-west.apache.org (localhost [127.0.0.1]) by svn01-us-west.apache.org (ASF Mail Server at svn01-us-west.apache.org) with ESMTP id D23973A0256 for ; Mon, 15 Feb 2016 05:36:59 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1730448 [1/12] - in /falcon/site/0.9: ./ css/ falconcli/ images/ images/logos/ images/profiles/ img/ js/ restapi/ Date: Mon, 15 Feb 2016 05:36:58 -0000 To: commits@falcon.apache.org From: pallavi@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20160215053659.D23973A0256@svn01-us-west.apache.org> Author: pallavi Date: Mon Feb 15 05:36:56 2016 New Revision: 1730448 URL: http://svn.apache.org/viewvc?rev=1730448&view=rev Log: Adding 0.9 Release docs html Added: falcon/site/0.9/ falcon/site/0.9/Appendix.html falcon/site/0.9/Architecture.png (with props) falcon/site/0.9/Configuration.html falcon/site/0.9/Distributed-mode.html falcon/site/0.9/Embedded-mode.html falcon/site/0.9/EntityDependency.png (with props) falcon/site/0.9/EntitySpecification.html falcon/site/0.9/FalconCLI.html falcon/site/0.9/FalconDocumentation.html falcon/site/0.9/FalconEmailNotification.html falcon/site/0.9/FalconNativeScheduler.html falcon/site/0.9/FeedSchedule.png (with props) falcon/site/0.9/HDFSDR.html falcon/site/0.9/HiveDR.html falcon/site/0.9/HiveIntegration.html falcon/site/0.9/ImportExport.html falcon/site/0.9/InstallationSteps.html falcon/site/0.9/MigrationInstructions.html falcon/site/0.9/OnBoarding.html falcon/site/0.9/Operability.html falcon/site/0.9/PrismSetup.png (with props) falcon/site/0.9/ProcessSchedule.png (with props) falcon/site/0.9/Recipes.html falcon/site/0.9/Security.html falcon/site/0.9/css/ falcon/site/0.9/css/apache-maven-fluido-1.3.0.min.css falcon/site/0.9/css/print.css falcon/site/0.9/css/site.css falcon/site/0.9/dependency-convergence.html falcon/site/0.9/dependency-info.html falcon/site/0.9/distribution-management.html falcon/site/0.9/falconcli/ falcon/site/0.9/falconcli/CommonCLI.html falcon/site/0.9/falconcli/ContinueInstance.html falcon/site/0.9/falconcli/Definition.html falcon/site/0.9/falconcli/DeleteEntity.html falcon/site/0.9/falconcli/DependencyEntity.html falcon/site/0.9/falconcli/DependencyInstance.html falcon/site/0.9/falconcli/EdgeMetadata.html falcon/site/0.9/falconcli/FalconCLI.html falcon/site/0.9/falconcli/FeedInstanceListing.html falcon/site/0.9/falconcli/HelpAdmin.html falcon/site/0.9/falconcli/KillInstance.html falcon/site/0.9/falconcli/LifeCycleInstance.html falcon/site/0.9/falconcli/LineageMetadata.html falcon/site/0.9/falconcli/ListEntity.html falcon/site/0.9/falconcli/ListInstance.html falcon/site/0.9/falconcli/ListMetadata.html falcon/site/0.9/falconcli/LogsInstance.html falcon/site/0.9/falconcli/Lookup.html falcon/site/0.9/falconcli/ParamsInstance.html falcon/site/0.9/falconcli/RelationMetadata.html falcon/site/0.9/falconcli/RerunInstance.html falcon/site/0.9/falconcli/ResumeEntity.html falcon/site/0.9/falconcli/ResumeInstance.html falcon/site/0.9/falconcli/RunningInstance.html falcon/site/0.9/falconcli/SLAAlert.html falcon/site/0.9/falconcli/Schedule.html falcon/site/0.9/falconcli/StatusAdmin.html falcon/site/0.9/falconcli/StatusEntity.html falcon/site/0.9/falconcli/StatusInstance.html falcon/site/0.9/falconcli/Submit.html falcon/site/0.9/falconcli/SubmitRecipe.html falcon/site/0.9/falconcli/SummaryEntity.html falcon/site/0.9/falconcli/SummaryInstance.html falcon/site/0.9/falconcli/SuspendEntity.html falcon/site/0.9/falconcli/SuspendInstance.html falcon/site/0.9/falconcli/Touch.html falcon/site/0.9/falconcli/TriageInstance.html falcon/site/0.9/falconcli/UpdateEntity.html falcon/site/0.9/falconcli/VersionAdmin.html falcon/site/0.9/falconcli/VertexEdgesMetadata.html falcon/site/0.9/falconcli/VertexMetadata.html falcon/site/0.9/falconcli/VerticesMetadata.html falcon/site/0.9/images/ falcon/site/0.9/images/accessories-text-editor.png (with props) falcon/site/0.9/images/add.gif (with props) falcon/site/0.9/images/apache-incubator-logo.png (with props) falcon/site/0.9/images/apache-maven-project-2.png (with props) falcon/site/0.9/images/application-certificate.png (with props) falcon/site/0.9/images/contact-new.png (with props) falcon/site/0.9/images/document-properties.png (with props) falcon/site/0.9/images/drive-harddisk.png (with props) falcon/site/0.9/images/falcon-logo.png (with props) falcon/site/0.9/images/fix.gif (with props) falcon/site/0.9/images/icon_error_sml.gif (with props) falcon/site/0.9/images/icon_help_sml.gif (with props) falcon/site/0.9/images/icon_info_sml.gif (with props) falcon/site/0.9/images/icon_success_sml.gif (with props) falcon/site/0.9/images/icon_warning_sml.gif (with props) falcon/site/0.9/images/image-x-generic.png (with props) falcon/site/0.9/images/internet-web-browser.png (with props) falcon/site/0.9/images/logos/ falcon/site/0.9/images/logos/build-by-maven-black.png (with props) falcon/site/0.9/images/logos/build-by-maven-white.png (with props) falcon/site/0.9/images/logos/maven-feather.png (with props) falcon/site/0.9/images/network-server.png (with props) falcon/site/0.9/images/package-x-generic.png (with props) falcon/site/0.9/images/profiles/ falcon/site/0.9/images/profiles/pre-release.png (with props) falcon/site/0.9/images/profiles/retired.png (with props) falcon/site/0.9/images/profiles/sandbox.png (with props) falcon/site/0.9/images/remove.gif (with props) falcon/site/0.9/images/rss.png (with props) falcon/site/0.9/images/update.gif (with props) falcon/site/0.9/images/window-new.png (with props) falcon/site/0.9/img/ falcon/site/0.9/img/glyphicons-halflings-white.png (with props) falcon/site/0.9/img/glyphicons-halflings.png (with props) falcon/site/0.9/index.html falcon/site/0.9/integration.html falcon/site/0.9/issue-tracking.html falcon/site/0.9/js/ falcon/site/0.9/js/apache-maven-fluido-1.3.0.min.js falcon/site/0.9/license.html falcon/site/0.9/mail-lists.html falcon/site/0.9/plugin-management.html falcon/site/0.9/plugins.html falcon/site/0.9/project-info.html falcon/site/0.9/project-summary.html falcon/site/0.9/restapi/ falcon/site/0.9/restapi/AdjacentVertices.html falcon/site/0.9/restapi/AdminConfig.html falcon/site/0.9/restapi/AdminStack.html falcon/site/0.9/restapi/AdminVersion.html falcon/site/0.9/restapi/AllEdges.html falcon/site/0.9/restapi/AllVertices.html falcon/site/0.9/restapi/Edge.html falcon/site/0.9/restapi/EntityDefinition.html falcon/site/0.9/restapi/EntityDelete.html falcon/site/0.9/restapi/EntityDependencies.html falcon/site/0.9/restapi/EntityLineage.html falcon/site/0.9/restapi/EntityList.html falcon/site/0.9/restapi/EntityResume.html falcon/site/0.9/restapi/EntitySchedule.html falcon/site/0.9/restapi/EntityStatus.html falcon/site/0.9/restapi/EntitySubmit.html falcon/site/0.9/restapi/EntitySubmitAndSchedule.html falcon/site/0.9/restapi/EntitySummary.html falcon/site/0.9/restapi/EntitySuspend.html falcon/site/0.9/restapi/EntityTouch.html falcon/site/0.9/restapi/EntityUpdate.html falcon/site/0.9/restapi/EntityValidate.html falcon/site/0.9/restapi/FeedInstanceListing.html falcon/site/0.9/restapi/FeedLookup.html falcon/site/0.9/restapi/FeedSLA.html falcon/site/0.9/restapi/Graph.html falcon/site/0.9/restapi/InstanceDependencies.html falcon/site/0.9/restapi/InstanceKill.html falcon/site/0.9/restapi/InstanceList.html falcon/site/0.9/restapi/InstanceLogs.html falcon/site/0.9/restapi/InstanceParams.html falcon/site/0.9/restapi/InstanceRerun.html falcon/site/0.9/restapi/InstanceResume.html falcon/site/0.9/restapi/InstanceRunning.html falcon/site/0.9/restapi/InstanceStatus.html falcon/site/0.9/restapi/InstanceSummary.html falcon/site/0.9/restapi/InstanceSuspend.html falcon/site/0.9/restapi/MetadataList.html falcon/site/0.9/restapi/MetadataRelations.html falcon/site/0.9/restapi/ResourceList.html falcon/site/0.9/restapi/Triage.html falcon/site/0.9/restapi/Vertex.html falcon/site/0.9/restapi/VertexProperties.html falcon/site/0.9/restapi/Vertices.html falcon/site/0.9/source-repository.html falcon/site/0.9/team-list.html Added: falcon/site/0.9/Appendix.html URL: http://svn.apache.org/viewvc/falcon/site/0.9/Appendix.html?rev=1730448&view=auto ============================================================================== --- falcon/site/0.9/Appendix.html (added) +++ falcon/site/0.9/Appendix.html Mon Feb 15 05:36:56 2016 @@ -0,0 +1,136 @@ + + + + + + + + + Falcon - Compatibility + + + + + + + + + + + + + + + + + +
+ + + + + + +
+ +
+

Compatibility

+
+

0.6-incubating Version Compatibility Matrix

+

+
    +
  • Hadoop 2.5.0 and above
    • +
  • Oozie 4.0.0 and above
    • +
  • Hive 0.11.0 and above
    • +
  • HCatalog 0.11.0 and above
    • +
  • Active MQ 5.4.3 and above
    • +
  • Titan 0.4.2 and above but below 0.5
+
+

0.6-incubating Tested Compatibility

+

+
    +
  • Hadoop 2.6.0
    • +
  • Oozie 4.1.0
    • +
  • Hive 0.14.0
    • +
  • HCatalog 0.14.0
    • +
  • Active MQ 5.4.3
    • +
  • Titan 0.4.2
    • +
  • Java 1.6, Java 1.7
+

Note : Oozie versions below 4.1.0 are not compatible with Java 1.7

+
+

0.6-incubating Release Notes

+

Major additions are listed below. Refer to CHANGES.txt for detailed issues addressed in this release.

+

+
    +
  • Security - Authorization, SSL
    • +
  • Lineage - More complete with better API
    • +
  • Recipes
    • +
  • Usability improvements - Dry run, entity summary, Pagination, etc.
    • +
  • Operability - Alerts, Audits, etc.
    • +
  • Refactoring - Messaging, Orchestration of workflows, etc.
    • +
  • Extension points for developers
    • +
  • Many bug fixes
+
+

0.6-incubating Upgrade Instructions

+

Please follow these instructions when upgrading from an older release.

+
+

Upgrading from 0.5-incubating

+

0.6-incubating is backwards incompatible with 0.5-incubating. It is recommended that user do not migrate from 0.5 to 0.6. However if the user must migrate from 0.5-incubating to 0.6-incubating, user should [[https://cwiki.apache.org/confluence/display/FALCON/Index][follow these instructions]

+
+

Upgrading from 0.4-incubating

+

It is not possible to upgrade to 0.6-incubating from 0.4-incubating.

+
+

Upgrading from 0.3-incubating

+

It is not possible to upgrade to 0.6-incubating from 0.3-incubating.

+
+
+ +
+ + + + Added: falcon/site/0.9/Architecture.png URL: http://svn.apache.org/viewvc/falcon/site/0.9/Architecture.png?rev=1730448&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/site/0.9/Architecture.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream Added: falcon/site/0.9/Configuration.html URL: http://svn.apache.org/viewvc/falcon/site/0.9/Configuration.html?rev=1730448&view=auto ============================================================================== --- falcon/site/0.9/Configuration.html (added) +++ falcon/site/0.9/Configuration.html Mon Feb 15 05:36:56 2016 @@ -0,0 +1,193 @@ + + + + + + + + + Falcon - Configuring Falcon + + + + + + + + + + + + + + + + + +
+ + + + + + +
+ +
+

Configuring Falcon

+

By default config directory used by falcon is {package dir}/conf. To override this (to use the same conf with multiple falcon upgrades), set environment variable FALCON_CONF to the path of the conf dir.

+

falcon-env.sh has been added to the falcon conf. This file can be used to set various environment variables that you need for you services. In addition you can set any other environment variables you might need. This file will be sourced by falcon scripts before any commands are executed. The following environment variables are available to set.

+
+
+# The java implementation to use. If JAVA_HOME is not found we expect java and jar to be in path
+#export JAVA_HOME=
+
+# any additional java opts you want to set. This will apply to both client and server operations
+#export FALCON_OPTS=
+
+# any additional java opts that you want to set for client only
+#export FALCON_CLIENT_OPTS=
+
+# java heap size we want to set for the client. Default is 1024MB
+#export FALCON_CLIENT_HEAP=
+
+# any additional opts you want to set for prism service.
+#export FALCON_PRISM_OPTS=
+
+# java heap size we want to set for the prism service. Default is 1024MB
+#export FALCON_PRISM_HEAP=
+
+# any additional opts you want to set for falcon service.
+#export FALCON_SERVER_OPTS=
+
+# java heap size we want to set for the falcon server. Default is 1024MB
+#export FALCON_SERVER_HEAP=
+
+# What is is considered as falcon home dir. Default is the base location of the installed software
+#export FALCON_HOME_DIR=
+
+# Where log files are stored. Default is logs directory under the base install location
+#export FALCON_LOG_DIR=
+
+# Where pid files are stored. Default is logs directory under the base install location
+#export FALCON_PID_DIR=
+
+# where the falcon active mq data is stored. Default is logs/data directory under the base install location
+#export FALCON_DATA_DIR=
+
+# Where do you want to expand the war file. By Default it is in /server/webapp dir under the base install dir.
+#export FALCON_EXPANDED_WEBAPP_DIR=
+
+
+
+

Advanced Configurations

+
+

Configuring Monitoring plugin to register catalog partitions

+

Falcon comes with a monitoring plugin that registers catalog partition. This comes in really handy during migration from filesystem based feeds to hcatalog based feeds. This plugin enables the user to de-couple the partition registration and assume that all partitions are already on hcatalog even before the migration, simplifying the hcatalog migration.

+

By default this plugin is disabled. To enable this plugin and leverage the feature, there are 3 pre-requisites:

+
+
+In {package dir}/conf/startup.properties, add
+*.workflow.execution.listeners=org.apache.falcon.catalog.CatalogPartitionHandler
+
+In the cluster definition, ensure registry endpoint is defined.
+Ex:
+<interface type="registry" endpoint="thrift://localhost:1109" version="0.13.3"/>
+
+In the feed definition, ensure the corresponding catalog table is mentioned in feed-properties
+Ex:
+<properties>
+    <property name="catalog.table" value="catalog:default:in_table#year={YEAR};month={MONTH};day={DAY};hour={HOUR};
+    minute={MINUTE}"/>
+</properties>
+
+
+

NOTE : for Mac OS users

+
+
+If you are using a Mac OS, you will need to configure the FALCON_SERVER_OPTS (explained above).
+
+In  {package dir}/conf/falcon-env.sh uncomment the following line
+#export FALCON_SERVER_OPTS=
+
+and change it to look as below
+export FALCON_SERVER_OPTS="-Djava.awt.headless=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="
+
+
+
+

Activemq

+

* falcon server starts embedded active mq. To control this behaviour, set the following system properties using -D option in environment variable FALCON_OPTS:

+
    +
  • falcon.embeddedmq=<true/false> - Should server start embedded active mq, default true
    • +
  • falcon.embeddedmq.port=<port> - Port for embedded active mq, default 61616
    • +
  • falcon.embeddedmq.data=<path> - Data path for embedded active mq, default {package dir}/logs/data
+
+

Falcon System Notifications

+

Some Falcon features such as late data handling, retries, metadata service, depend on JMS notifications sent when the Oozie workflow completes. These system notifications are sent as part of Falcon Post Processing action. Given that the post processing action is also a job, it is prone to failures and in case of failures, Falcon is blind to the status of the workflow. To alleviate this problem and make the notifications more reliable, you can enable Oozie's JMS notification feature and disable Falcon post-processing notification by making the following changes:

+
    +
  • In Falcon runtime.properties, set *.falcon.jms.notification.enabled to false. This will turn off JMS notification in post-processing.
    • +
  • Copy notification related properties in oozie/conf/oozie-site.xml to oozie-site.xml of the Oozie installation. Restart Oozie so changes get reflected.
+

NOTE : If you disable Falcon post-processing JMS notification and not enable Oozie JMS notification, features such as failure retry, late data handling and metadata service will be disabled for all entities on the server.

+
+

Enabling Falcon Native Scheudler

+

$FALCON_HOME/conf/startup.properties before starting the Falcon Server. For details on the same, refer to Falcon Native Scheduler

+
+

Adding Extension Libraries

+

Library extensions allows users to add custom libraries to entity lifecycles such as feed retention, feed replication and process execution. This is useful for usecases such as adding filesystem extensions. To enable this, add the following configs to startup.properties: *.libext.paths=<paths to be added to all entity lifecycles>

+

*.libext.feed.paths=<paths to be added to all feed lifecycles>

+

*.libext.feed.retentions.paths=<paths to be added to feed retention workflow>

+

*.libext.feed.replication.paths=<paths to be added to feed replication workflow>

+

*.libext.process.paths=<paths to be added to process workflow>

+

The configured jars are added to falcon classpath and the corresponding workflows.

+
+
+ +
+ + + + Added: falcon/site/0.9/Distributed-mode.html URL: http://svn.apache.org/viewvc/falcon/site/0.9/Distributed-mode.html?rev=1730448&view=auto ============================================================================== --- falcon/site/0.9/Distributed-mode.html (added) +++ falcon/site/0.9/Distributed-mode.html Mon Feb 15 05:36:56 2016 @@ -0,0 +1,250 @@ + + + + + + + + + Falcon - Distributed Mode + + + + + + + + + + + + + + + + + +
+ + + + + + +
+ +
+

Distributed Mode

+

Following are the steps needed to package and deploy Falcon in Embedded Mode. You need to complete Steps 1-3 mentioned here before proceeding further.

+
+

Package Falcon

+

Ensure that you are in the base directory (where you cloned Falcon). Let’s call it {project dir}

+
+
+$mvn clean assembly:assembly -DskipTests -DskipCheck=true -Pdistributed,hadoop-2
+
+
+
+
+$ls {project dir}/target/
+
+
+

It should give an output like below :

+
+
+apache-falcon-distributed-${project.version}-server.tar.gz
+apache-falcon-distributed-${project.version}-sources.tar.gz
+archive-tmp
+maven-shared-archive-resources
+
+
+

+
    +
  • apache-falcon-distributed-${project.version}-sources.tar.gz contains source files of Falcon repo.
+

+
    +
  • apache-falcon-distributed-${project.version}-server.tar.gz package contains project artifacts along with it's
dependencies, configuration files and scripts required to deploy Falcon. +

Tar can be found in {project dir}/target/apache-falcon-distributed-${project.version}-server.tar.gz . This is the tar used for installing Falcon. Lets call it {falcon package}

+

Tar is structured as follows.

+
+
+
+|- bin
+   |- falcon
+   |- falcon-start
+   |- falcon-stop
+   |- falcon-status
+   |- falcon-config.sh
+   |- service-start.sh
+   |- service-stop.sh
+   |- service-status.sh
+   |- prism-stop
+   |- prism-start
+   |- prism-status
+|- conf
+   |- startup.properties
+   |- runtime.properties
+   |- client.properties
+   |- prism.keystore
+   |- log4j.xml
+   |- falcon-env.sh
+|- docs
+|- client
+   |- lib (client support libs)
+|- server
+   |- webapp
+      |- falcon.war
+      |- prism.war
+|- oozie
+   |- conf
+   |- libext
+|- hadooplibs
+|- README
+|- NOTICE.txt
+|- LICENSE.txt
+|- DISCLAIMER.txt
+|- CHANGES.txt
+
+
+
+

Installing & running Falcon

+
+

Installing Falcon

+

Running Falcon in distributed mode requires bringing up both prism and server.As the name suggests Falcon prism splits the request it gets to the Falcon servers. It is a good practice to start prism and server with their corresponding configurations separately. Create separate directory for prism and server. Let's call them {falcon-prism-dir} and {falcon-server-dir} respectively.

+

For prism

+
+
+$mkdir {falcon-prism-dir}
+$tar -xzvf {falcon package}
+
+
+

For server

+
+
+$mkdir {falcon-server-dir}
+$tar -xzvf {falcon package}
+
+
+
+

Starting Prism

+
+
+cd {falcon-prism-dir}/falcon-distributed-${project.version}
+bin/prism-start [-port <port>]
+
+
+

By default, * prism server starts at port 16443. To change the port, use -port option

+

* falcon.enableTLS can be set to true or false explicitly to enable SSL, if not port that end with 443 will automatically put prism on https://

+

* prism starts with conf from {falcon-prism-dir}/falcon-distributed-${project.version}/conf. To override this (to use the same conf with multiple prism upgrades), set environment variable FALCON_CONF to the path of conf dir. You can find the instructions for configuring Falcon here.

+

Enabling prism-client *If prism is not started using default-port 16443 then edit the following property in {falcon-prism-dir}/falcon-distributed-${project.version}/conf/client.properties falcon.url=http://{machine-ip}:{prism-port}/

+
+

Starting Falcon Server

+
+
+$cd {falcon-server-dir}/falcon-distributed-${project.version}
+$bin/falcon-start [-port <port>]
+
+
+

By default, * If falcon.enableTLS is set to true explicitly or not set at all, Falcon starts at port 15443 on https:// by default.

+

* If falcon.enableTLS is set to false explicitly, Falcon starts at port 15000 on http://.

+

* To change the port, use -port option.

+

* If falcon.enableTLS is not set explicitly, port that ends with 443 will automatically put Falcon on https://. Any other port will put Falcon on http://.

+

* server starts with conf from {falcon-server-dir}/falcon-distributed-${project.version}/conf. To override this (to use the same conf with multiple server upgrades), set environment variable FALCON_CONF to the path of conf dir. You can find the instructions for configuring Falcon here.

+

Enabling server-client *If server is not started using default-port 15443 then edit the following property in {falcon-server-dir}/falcon-distributed-${project.version}/conf/client.properties. You can find the instructions for configuring Falcon here. falcon.url=http://{machine-ip}:{server-port}/

+

NOTE : https is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. By default Falcon runs in https mode. But user can configure it to http.

+
+

Using Falcon

+
+
+$cd {falcon-prism-dir}/falcon-distributed-${project.version}
+$bin/falcon admin -version
+Falcon server build version: {Version:"${project.version}-SNAPSHOT-rd7e2be9afa2a5dc96acd1ec9e325f39c6b2f17f7",
+Mode:"embedded"}
+
+$bin/falcon help
+(for more details about Falcon cli usage)
+
+
+
+

Dashboard

+

Once Falcon / prism is started, you can view the status of Falcon entities using the Web-based dashboard. You can open your browser at the corresponding port to use the web UI.

+

Falcon dashboard makes the REST api calls as user "falcon-dashboard". If this user does not exist on your Falcon and Oozie servers, please create the user.

+
+
+## create user.
+[root@falconhost ~] useradd -U -m falcon-dashboard -G users
+
+## verify user is created with membership in correct groups.
+[root@falconhost ~] groups falcon-dashboard
+falcon-dashboard : falcon-dashboard users
+[root@falconhost ~]
+
+
+
+

Stopping Falcon Server

+
+
+$cd {falcon-server-dir}/falcon-distributed-${project.version}
+$bin/falcon-stop
+
+
+
+

Stopping Falcon Prism

+
+
+$cd {falcon-prism-dir}/falcon-distributed-${project.version}
+$bin/prism-stop
+
+
+
+
+ +
+ + + + Added: falcon/site/0.9/Embedded-mode.html URL: http://svn.apache.org/viewvc/falcon/site/0.9/Embedded-mode.html?rev=1730448&view=auto ============================================================================== --- falcon/site/0.9/Embedded-mode.html (added) +++ falcon/site/0.9/Embedded-mode.html Mon Feb 15 05:36:56 2016 @@ -0,0 +1,276 @@ + + + + + + + + + Falcon - Embedded Mode + + + + + + + + + + + + + + + + + +
+ + + + + + +
+ +
+

Embedded Mode

+

Following are the steps needed to package and deploy Falcon in Embedded Mode. You need to complete Steps 1-3 mentioned here before proceeding further.

+
+

Package Falcon

+

Ensure that you are in the base directory (where you cloned Falcon). Let’s call it {project dir}

+
+
+$mvn clean assembly:assembly -DskipTests -DskipCheck=true
+
+
+
+
+$ls {project dir}/target/
+
+
+

It should give an output like below :

+
+
+apache-falcon-${project.version}-bin.tar.gz
+apache-falcon-${project.version}-sources.tar.gz
+archive-tmp
+maven-shared-archive-resources
+
+
+

* apache-falcon-${project.version}-sources.tar.gz contains source files of Falcon repo.

+

* apache-falcon-${project.version}-bin.tar.gz package contains project artifacts along with it's dependencies, configuration files and scripts required to deploy Falcon.

+

Tar can be found in {project dir}/target/apache-falcon-${project.version}-bin.tar.gz

+

Tar is structured as follows :

+
+
+
+|- bin
+   |- falcon
+   |- falcon-start
+   |- falcon-stop
+   |- falcon-status
+   |- falcon-config.sh
+   |- service-start.sh
+   |- service-stop.sh
+   |- service-status.sh
+|- conf
+   |- startup.properties
+   |- runtime.properties
+   |- prism.keystore
+   |- client.properties
+   |- log4j.xml
+   |- falcon-env.sh
+|- docs
+|- client
+   |- lib (client support libs)
+|- server
+   |- webapp
+      |- falcon.war
+|- data
+   |- falcon-store
+   |- graphdb
+   |- localhost
+|- examples
+   |- app
+      |- hive
+      |- oozie-mr
+      |- pig
+   |- data
+   |- entity
+      |- filesystem
+      |- hcat
+|- oozie
+   |- conf
+   |- libext
+|- logs
+|- hadooplibs
+|- README
+|- NOTICE.txt
+|- LICENSE.txt
+|- DISCLAIMER.txt
+|- CHANGES.txt
+
+
+
+

Installing & running Falcon

+

Running Falcon in embedded mode requires bringing up server.

+
+
+$tar -xzvf {falcon package}
+$cd falcon-${project.version}
+
+
+
+

Starting Falcon Server

+
+
+$cd falcon-${project.version}
+$bin/falcon-start [-port <port>]
+
+
+

By default, * If falcon.enableTLS is set to true explicitly or not set at all, Falcon starts at port 15443 on https:// by default.

+

* If falcon.enableTLS is set to false explicitly, Falcon starts at port 15000 on http://.

+

* To change the port, use -port option.

+

* If falcon.enableTLS is not set explicitly, port that ends with 443 will automatically put Falcon on https://. Any other port will put Falcon on http://.

+

* Server starts with conf from {falcon-server-dir}/falcon-distributed-${project.version}/conf. To override this (to use the same conf with multiple server upgrades), set environment variable FALCON_CONF to the path of conf dir. You can find the instructions for configuring Falcon here.

+
+

Enabling server-client

+

If server is not started using default-port 15443 then edit the following property in {falcon-server-dir}/falcon-${project.version}/conf/client.properties

+

falcon.url=http://{machine-ip}:{server-port}/

+
+

Using Falcon

+
+
+$cd falcon-${project.version}
+$bin/falcon admin -version
+Falcon server build version: {Version:"${project.version}-SNAPSHOT-rd7e2be9afa2a5dc96acd1ec9e325f39c6b2f17f7",Mode:
+"embedded",Hadoop:"${hadoop.version}"}
+
+$bin/falcon help
+(for more details about Falcon cli usage)
+
+
+

Note : https is the secure version of HTTP, the protocol over which data is sent between your browser and the website that you are connected to. By default Falcon runs in https mode. But user can configure it to http.

+
+

Dashboard

+

Once Falcon server is started, you can view the status of Falcon entities using the Web-based dashboard. You can open your browser at the corresponding port to use the web UI.

+

Falcon dashboard makes the REST api calls as user "falcon-dashboard". If this user does not exist on your Falcon and Oozie servers, please create the user.

+
+
+## create user.
+[root@falconhost ~] useradd -U -m falcon-dashboard -G users
+
+## verify user is created with membership in correct groups.
+[root@falconhost ~] groups falcon-dashboard
+falcon-dashboard : falcon-dashboard users
+[root@falconhost ~]
+
+
+
+

Running Examples using embedded package

+
+
+$cd falcon-${project.version}
+$bin/falcon-start
+
+
+

Make sure the Hadoop and Oozie endpoints are according to your setup in examples/entity/filesystem/standalone-cluster.xml The cluster locations,staging and working dirs, MUST be created prior to submitting a cluster entity to Falcon. staging must have 777 permissions and the parent dirs must have execute permissions working must have 755 permissions and the parent dirs must have execute permissions

+
+
+$bin/falcon entity -submit -type cluster -file examples/entity/filesystem/standalone-cluster.xml
+
+
+

Submit input and output feeds:

+
+
+$bin/falcon entity -submit -type feed -file examples/entity/filesystem/in-feed.xml
+$bin/falcon entity -submit -type feed -file examples/entity/filesystem/out-feed.xml
+
+
+

Set-up workflow for the process:

+
+
+$hadoop fs -put examples/app /
+
+
+

Submit and schedule the process:

+
+
+$bin/falcon entity -submitAndSchedule -type process -file examples/entity/filesystem/oozie-mr-process.xml
+$bin/falcon entity -submitAndSchedule -type process -file examples/entity/filesystem/pig-process.xml
+
+
+

Generate input data:

+
+
+$examples/data/generate.sh <<hdfs endpoint>>
+
+
+

Get status of instances:

+
+
+$bin/falcon instance -status -type process -name oozie-mr-process -start 2013-11-15T00:05Z -end 2013-11-15T01:00Z
+
+
+

HCat based example entities are in examples/entity/hcat.

+
+

Stopping Falcon Server

+
+
+$cd falcon-${project.version}
+$bin/falcon-stop
+
+
+
+
+ +
+ + + + Added: falcon/site/0.9/EntityDependency.png URL: http://svn.apache.org/viewvc/falcon/site/0.9/EntityDependency.png?rev=1730448&view=auto ============================================================================== Binary file - no diff available. Propchange: falcon/site/0.9/EntityDependency.png ------------------------------------------------------------------------------ svn:mime-type = application/octet-stream