falcon-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pall...@apache.org
Subject falcon git commit: FALCON-2078 Documentation for Process SLA
Date Wed, 19 Oct 2016 11:00:58 GMT
Repository: falcon
Updated Branches:
  refs/heads/master 145f9bc77 -> f191ea540


FALCON-2078 Documentation for Process SLA

Author: Praveen Adlakha <adlakha.praveen@gmail.com>

Reviewers: @sepreece, @pallavi-rao

Closes #281 from PraveenAdlakha/2078 and squashes the following commits:

1dcf8e1 [Praveen Adlakha] comments addressed
e37832e [Praveen Adlakha] comments addressed
94bdcc0 [Praveen Adlakha] Falcon-2078 Documentation for Process SLA


Project: http://git-wip-us.apache.org/repos/asf/falcon/repo
Commit: http://git-wip-us.apache.org/repos/asf/falcon/commit/f191ea54
Tree: http://git-wip-us.apache.org/repos/asf/falcon/tree/f191ea54
Diff: http://git-wip-us.apache.org/repos/asf/falcon/diff/f191ea54

Branch: refs/heads/master
Commit: f191ea540e77fafa9b6e39aa9079e7dd375dfed8
Parents: 145f9bc
Author: Praveen Adlakha <adlakha.praveen@gmail.com>
Authored: Wed Oct 19 16:30:50 2016 +0530
Committer: Pallavi Rao <pallavi.rao@inmobi.com>
Committed: Wed Oct 19 16:30:50 2016 +0530

----------------------------------------------------------------------
 .../falcon/persistence/EntitySLAAlertBean.java  |  2 +-
 docs/src/site/twiki/EntitySLAAlerting.twiki     | 57 +++++++++++++++
 docs/src/site/twiki/EntitySLAMonitoring.twiki   | 25 +++++++
 docs/src/site/twiki/FalconDatabase.twiki        | 74 ++++++++++++++++++++
 docs/src/site/twiki/FalconDocumentation.twiki   |  2 +-
 docs/src/site/twiki/FeedSLAMonitoring.twiki     | 27 -------
 .../site/twiki/GraphiteMetricCollection.twiki   |  8 ++-
 docs/src/site/twiki/Operability.twiki           |  5 +-
 8 files changed, 167 insertions(+), 33 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java
----------------------------------------------------------------------
diff --git a/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java b/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java
index 1419f48..186c5e0 100644
--- a/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java
+++ b/common/src/main/java/org/apache/falcon/persistence/EntitySLAAlertBean.java
@@ -35,7 +35,7 @@ import javax.validation.constraints.NotNull;
 
 //SUSPEND CHECKSTYLE CHECK LineLengthCheck
 /**
- * Feed SLA monitoring.
+ * Entity SLA monitoring.
  * */
 @Entity
 @NamedQueries({

http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/EntitySLAAlerting.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/EntitySLAAlerting.twiki b/docs/src/site/twiki/EntitySLAAlerting.twiki
new file mode 100644
index 0000000..8534ba6
--- /dev/null
+++ b/docs/src/site/twiki/EntitySLAAlerting.twiki
@@ -0,0 +1,57 @@
+---++Entity SLA Alerting
+
+Falcon supports SLA in feed and process.
+
+Types of SLA supported for feed:
+
+   1.slaLow
+   1.slaHigh
+
+To know more about feedSla look at [[EntitySpecification][Feed Specification]]
+
+Types of SLA supported for process:
+
+   1.shouldStartIn
+   1.shouldEndIn
+
+To know more about processSla look at [[EntitySpecification][Process Specification]]
+
+Falcon Entity Alerting service do the following things:
+
+   1.Monitor instances of feed and process and send notifications to all the listeners attached
to it.
+   1.In case of feed it notifies when an *slaHigh* miss happens. slaLow is not supported.
+   1.In case of process it notifies when an SLA miss for *shouldEndIn* happens. shouldStartIn
is not supported.
+
+Entity SLA Alert service depends upon [[EntitySLAMonitoring][Falcon Entity SLA Monitoring]]
to know which process and feed instances are to be monitored.
+
+*How to attach listeners:*
+
+You can write custom listeners to do some action whenever a process or feed instance misses
its SLA.
+To attach listeners please add below property in startup.properties:
+
+<verbatim>
+
+*.entityAlert.listeners=org.apache.customPath.customListener
+
+</verbatim>
+
+Currently Falcon natively supports [[BacklogMetricEmitterService][Back Log Emitter Service]]
as a listener to EntitySLAAlert service
+
+---++Dependencies:
+
+*Other Services:*
+
+To enable Enity SLA Alerting service you need to enable [[EntitySLAMonitoring][Falcon Entity
SLA Monitoring]]
+
+ Following properties are needed in startup.properties:
+
+<verbatim>
+
+*.application.services=org.apache.falcon.service.EntitySLAAlertService
+
+*.entity.sla.statusCheck.frequency.seconds=600
+</verbatim>
+
+*Falcon Database:*
+
+Entity SLA Alerting service maintains its state in the database.It needs one table *ENTITY_SLA_ALERTS*
please have a look at [[FalconDatabase]] to know how to create it.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/EntitySLAMonitoring.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/EntitySLAMonitoring.twiki b/docs/src/site/twiki/EntitySLAMonitoring.twiki
new file mode 100644
index 0000000..bdd9ac4
--- /dev/null
+++ b/docs/src/site/twiki/EntitySLAMonitoring.twiki
@@ -0,0 +1,25 @@
+---++Falcon Entity SLA Monitoring
+
+Entity SLA monitoring allows you to monitor the entity (process and feed) .It keeps track
of the instances of the entity that are running and stores them in the db.
+
+
+---++Dependencies:
+
+*Other Services:*
+
+Entity SLA monitoring service requires FalconJPAService to be up. Following are the values
you need to set to run EntitySLAMonitoring.
+In startup.properties:
+
+<verbatim>
+*.application.services= org.apache.falcon.state.store.service.FalconJPAService,
+                        org.apache.falcon.service.EntitySLAMonitoringService
+</verbatim>
+
+
+*Falcon Database:*
+
+Entity SLA monitoring service maintains its state in the database.It needs two tables:
+
+   1.MONITORED_ENTITY
+   1.PENDING_INSTANCES
+please have a look at [[FalconDatabase]] to know how to create them.

http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/FalconDatabase.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/FalconDatabase.twiki b/docs/src/site/twiki/FalconDatabase.twiki
new file mode 100644
index 0000000..5800fa7
--- /dev/null
+++ b/docs/src/site/twiki/FalconDatabase.twiki
@@ -0,0 +1,74 @@
+---+++ Configuring the state store for Falcon
+You can configure statestore by making changes to __$FALCON_HOME/conf/statestore.properties__
as follows. You will need to restart Falcon Server for the changes to take effect.
+
+Falcon Server needs to maintain state of the entities and instances in a persistent store
for the system to be recoverable. Since Prism only federates, it does not need to maintain
any state information. Following properties need to be set in statestore.properties of Falcon
Servers:
+<verbatim>
+######### StateStore Properties #####
+*.falcon.state.store.impl=org.apache.falcon.state.store.jdbc.JDBCStateStore
+*.falcon.statestore.jdbc.driver=org.apache.derby.jdbc.EmbeddedDriver
+*.falcon.statestore.jdbc.url=jdbc:derby:data/falcon.db
+# StateStore credentials file where username,password and other properties can be stored
securely.
+# Set this credentials file permission 400 ;the user who starts falcon should only have read
permission.
+# Give Absolute path to credentials file along with file name or put in classpath with file
name statestore.credentials.
+# Credentials file should be present either in given location or class path, otherwise falcon
won't start.
+*.falcon.statestore.credentials.file=
+*.falcon.statestore.jdbc.username=sa
+*.falcon.statestore.jdbc.password=
+*.falcon.statestore.connection.data.source=org.apache.commons.dbcp.BasicDataSource
+# Maximum number of active connections that can be allocated from this pool at the same time.
+*.falcon.statestore.pool.max.active.conn=10
+*.falcon.statestore.connection.properties=
+# Indicates the interval (in milliseconds) between eviction runs.
+*.falcon.statestore.validate.db.connection.eviction.interval=300000
+## The number of objects to examine during each run of the idle object evictor thread.
+*.falcon.statestore.validate.db.connection.eviction.num=10
+## Creates Falcon DB.
+## If set to true, Falcon creates the DB schema if it does not exist. If the DB schema exists
is a NOP.
+## If set to false, Falcon does not create the DB schema. If the DB schema does not exist
it fails start up.
+*.falcon.statestore.create.db.schema=true
+</verbatim>
+
+The _*.falcon.statestore.jdbc.url_ property in statestore.properties determines the DB and
data location. All other properties are common across RDBMS.
+
+*NOTE : Although multiple Falcon Servers can share a DB (not applicable for Derby DB), it
is recommended that you have different DBs for different Falcon Servers for better performance.*
+
+You will need to create the state DB and tables before starting the Falcon Server. To create
tables, a tool comes bundled with the Falcon installation. You can use the _falcon-db.sh_
script to create tables in the DB. The script needs to be run only for Falcon Servers and
can be run by any user that has execute permission on the script. The script picks up the
DB connection details from __$FALCON_HOME/conf/statestore.properties__. Ensure that you have
granted the right privileges to the user mentioned in statestore.properties_, so the tables
can be created.
+
+You can use the help command to get details on the sub-commands supported:
+<verbatim>
+./bin/falcon-db.sh help
+usage:
+      Falcon DB initialization tool currently supports Derby DB/ Mysql
+
+      falcondb help : Display usage for all commands or specified command
+
+      falcondb version : Show Falcon DB version information
+
+      falcondb create <OPTIONS> : Create Falcon DB schema
+                      -run             Confirmation option regarding DB schema creation/upgrade
+                      -sqlfile <arg>   Generate SQL script instead of creating/upgrading
the DB
+                                       schema
+
+      falcondb upgrade <OPTIONS> : Upgrade Falcon DB schema
+                       -run             Confirmation option regarding DB schema creation/upgrade
+                       -sqlfile <arg>   Generate SQL script instead of creating/upgrading
the DB
+                                        schema
+
+</verbatim>
+Currently, MySQL, postgreSQL and Derby are supported as state stores. We may extend support
to other DBs in the future. Falcon has been tested against MySQL v5.5 and PostgreSQL v9.5.
If you are using MySQL ensure you also copy mysql-connector-java-<version>.jar under
__$FALCON_HOME/server/webapp/falcon/WEB-INF/lib__ and __$FALCON_HOME/client/lib__
+
+---++++ Using Derby as the State Store
+Using Derby is ideal for QA and staging setup. Falcon comes bundled with a Derby connector
and no explicit setup is required (although you can set it up) in terms creating the DB or
tables.
+For example,
+ <verbatim> *.falcon.statestore.jdbc.url=jdbc:derby:data/falcon.db;create=true </verbatim>
+
+ tells Falcon to use the Derby JDBC connector, with data directory, $FALCON_HOME/data/ and
DB name 'falcon'. If _create=true_ is specified, you will not need to create a DB up front;
a database will be created if it does not exist.
+
+---++++ Using MySQL as the State Store
+The jdbc.url property in statestore.properties determines the DB and data location.
+For example,
+ <verbatim> *.falcon.statestore.jdbc.url=jdbc:mysql://localhost:3306/falcon </verbatim>
+
+ tells Falcon to use the MySQL JDBC connector, which is accessible @localhost:3306, with
DB name 'falcon'.
+
+ Note: First time we have to manually create the schema in production as we have set falcon.statestore.create.db.schema
= false
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/FalconDocumentation.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/FalconDocumentation.twiki b/docs/src/site/twiki/FalconDocumentation.twiki
index fe1c0de..96d37a2 100644
--- a/docs/src/site/twiki/FalconDocumentation.twiki
+++ b/docs/src/site/twiki/FalconDocumentation.twiki
@@ -823,7 +823,7 @@ Extensions is detailed in [[Extensions][Extensions]].
 
 ---++ Monitoring
 
-Monitoring and Operationalizing Falcon is detailed in [[Operability][Operability]] and [[FeedSLAMonitoring][Falcon
Feed SLA Monitoring]].
+Monitoring and Operationalizing Falcon is detailed in [[Operability][Operability]] and [[EntitySLAMonitoring][Falcon
Entity SLA Monitoring]].
 
 ---++ Email Notification
 Notification for instance completion in Falcon is defined in [[FalconEmailNotification][Falcon
Email Notification]].

http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/FeedSLAMonitoring.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/FeedSLAMonitoring.twiki b/docs/src/site/twiki/FeedSLAMonitoring.twiki
deleted file mode 100644
index 469c0aa..0000000
--- a/docs/src/site/twiki/FeedSLAMonitoring.twiki
+++ /dev/null
@@ -1,27 +0,0 @@
----++Falcon Feed SLA Monitoring
-
-Feed SLA monitoring allows you to monitor the feed ,It keeps track of the instances of the
feed which are running and stores them in the db.
-
-Feed SLA monitoring service requires FalconJPAService to be up.Following are the values we
need to set to run FeedSLAMonitoring -
-In startup.properties :
-
-*.application.services= org.apache.falcon.state.store.service.FalconJPAService,
-                        org.apache.falcon.service.EntitySLAMonitoringService
-
-These properties are required for FalconJPAService in statestore.properties:
-
-   * *falcon.state.store.impl* - org.apache.falcon.state.store.jdbc.JDBCStateStore
-   * *falcon.statestore.jdbc.driver*  -org.apache.derby.jdbc.EmbeddedDriver
-   * *falcon.statestore.jdbc.url* -jdbc:derby:target/test-data/data.db;create=true
-   * *falcon.statestore.connection.data.source*   -org.apache.commons.dbcp.BasicDataSource
-   Maximum number of active connections that can be allocated from this pool at the same
time.
-   * *falcon.statestore.pool.max.active.conn* -10
-   * *falcon.statestore.connection.properties* -
-   Indicates the interval (in milliseconds) between eviction runs.
-   * *falcon.statestore.validate.db.connection.eviction.interval* -300000
-   The number of objects to examine during each run of the idle object evictor thread.
-   * *falcon.statestore.validate.db.connection.eviction.num*  -10
-   Creates Falcon DB. If set to true, it creates the DB schema if it does not exist. If the
DB schema exists is a NOP.If set to false, it does not create the DB schema. If the DB schema
does not exist it fails start up.
-   * *falcon.statestore.create.db.schema* -false
-
-Note: First time we have to manually create the schema in production as we have set falcon.statestore.create.db.schema
= false
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/GraphiteMetricCollection.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/GraphiteMetricCollection.twiki b/docs/src/site/twiki/GraphiteMetricCollection.twiki
index c76e68d..0ae0498 100644
--- a/docs/src/site/twiki/GraphiteMetricCollection.twiki
+++ b/docs/src/site/twiki/GraphiteMetricCollection.twiki
@@ -8,15 +8,17 @@ Graphite Metric Collection currently allows to collect the following metrics
at
 
 To send data to graphite we need to intialize metricNotificationService in startup.properties:
 
+<verbatim>
 *.application.services= org.apache.falcon.metrics.MetricNotificationService,
-
+</verbatim>
 
 Add following properties for graphiteNotificationPlugin :
 
-Graphite properties
+*Graphite properties*
+<verbatim>
    * *.falcon.graphite.hostname=localhost
    * *.falcon.graphite.port=2003
    * *.falcon.graphite.frequency=1
    * *.falcon.graphite.prefix=falcon
-
+</verbatim>
 The falcon.graphite.frequency is in seconds and all the time that is being sent to graphite
is in seconds.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/falcon/blob/f191ea54/docs/src/site/twiki/Operability.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Operability.twiki b/docs/src/site/twiki/Operability.twiki
index 2bccb51..f01c235 100644
--- a/docs/src/site/twiki/Operability.twiki
+++ b/docs/src/site/twiki/Operability.twiki
@@ -207,13 +207,16 @@ public class FalconUserJMSClient {
 
 ---++ Alerts
 
-Falcon generates alerts for unrecoverable errors into a log file by default.
+Falcon generates two type of alerts:
+
+1. By default it logs unrecoverable errors into a log file
 Users can view these alerts in the alerts.log file, by default this file is created
 under ${user.dir}/logs/ directory.
 
 Users may also extend the Falcon Alerting plugin to send events to systems like Nagios, etc.
by
 extending org.apache.falcon.plugin.AlertingPlugin interface.
 
+2. Alerts on *SLA misses* for feeds and process is detailed in [[EntitySLAAlerting][Entity
SLA Alerting]].
 
 ---++ Audits
 


Mime
View raw message