eagle-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From h..@apache.org
Subject eagle git commit: [MINOR][DOC] Add oozie applicaton installation doc
Date Wed, 05 Apr 2017 05:50:59 GMT
Repository: eagle
Updated Branches:
  refs/heads/master ee55054a7 -> 0c52711b8


[MINOR][DOC] Add oozie applicaton installation doc

Author: chitin <chitin1027@gmail.com>

Closes #904 from chitin/oozie-doc.


Project: http://git-wip-us.apache.org/repos/asf/eagle/repo
Commit: http://git-wip-us.apache.org/repos/asf/eagle/commit/0c52711b
Tree: http://git-wip-us.apache.org/repos/asf/eagle/tree/0c52711b
Diff: http://git-wip-us.apache.org/repos/asf/eagle/diff/0c52711b

Branch: refs/heads/master
Commit: 0c52711b897a4fd6d5aaf7330e89572582f1d189
Parents: ee55054
Author: chitin <chitin1027@gmail.com>
Authored: Wed Apr 5 13:50:48 2017 +0800
Committer: Hao Chen <hchen9@ebay.com>
Committed: Wed Apr 5 13:50:48 2017 +0800

----------------------------------------------------------------------
 docs/docs/applications.md                    |  63 ++++++++++++++++++++++
 docs/docs/include/images/oozie_audit_log.png | Bin 0 -> 41562 bytes
 docs/docs/include/images/oozie_install_1.png | Bin 0 -> 165098 bytes
 docs/docs/include/images/oozie_install_2.png | Bin 0 -> 153179 bytes
 docs/docs/include/images/oozie_install_3.png | Bin 0 -> 94200 bytes
 docs/docs/include/images/oozie_policy_1.png  | Bin 0 -> 101463 bytes
 6 files changed, 63 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/applications.md
----------------------------------------------------------------------
diff --git a/docs/docs/applications.md b/docs/docs/applications.md
index 74efcc6..a54c236 100644
--- a/docs/docs/applications.md
+++ b/docs/docs/applications.md
@@ -70,6 +70,69 @@ The example policy monitors the 'delete' operation on hosts in 'SECURITY'
zone.
 ## Questions on this application
 
 ---
+# OOZIE Data Activity Monitoring
+ 
+## Monitor Requirements
+
+This application aims to monitor user activities on OOZIE via the oozie audit log. Once any
abnormal user activity is detected, an alert is sent in several seconds. The whole pipeline
of this application is
+
+* Kafka ingest: this application consumes data from Kafka. In other words, users have to
stream the log into Kafka first. 
+
+* Data re-procesing, which includes raw log parser, sensitivity information joiner. 
+
+* Kafka sink: parsed data will flows into Kafka again, which will be consumed by the alert
engine. 
+
+* Policy evaluation: the alert engine (hosted in Alert Engine app) evaluates each data event
to check if the data violate the user defined policy. An alert is generated if the data matches
the policy.
+
+![OOZIEAUDITLOG](include/images/oozie_audit_log.png)
+
+
+## Setup & Installation
+
+* Choose a site to install this application. For example 'sandbox'
+
+* Install "Oozie Audit Log Monitor" app step by step
+
+    ![Install Step 2](include/images/oozie_install_1.png)
+
+    ![Install Step 3](include/images/oozie_install_2.png)
+
+    ![Install Step 4](include/images/oozie_install_3.png)
+
+
+## How to collect the log
+
+To collect the raw audit log on namenode servers, a log collector is needed. Users can choose
any tools they like. There are some common solutions available: [logstash](https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html),
[filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-getting-started.html),
log4j appender, etcs. 
+
+For detailed instruction, refer to: [How to stream audit log into Kafka](using-eagle/#how-to-stream-audit-log-into-kafka)
+
+## Sample policies
+
+### 1. monitor file/folder operations 
+
+Delete a file/folder on HDFS. 
+
+```
+from OOZIE_AUDIT_LOG_STREAM_SANDBOX[ip=='localhost'] select * insert into oozie_audit_log_enriched_stream_out
+```
+
+OOZIE_AUDIT_LOG_STREAM_SANDBOX is the input stream name, and oozie_audit_log_enriched_stream_out
is the output stream name, the content between [] is the monitoring conditions. `user`, `ip`,
`jobId`, `operation` and `timestamp` is the fields of oozie audit logs.
+
+   ![Policy 1](include/images/oozie_policy_1.png)
+
+### 2. classify the file/folder on OOZIE
+
+Users may want to mark some jobId on OOZIE as sensitive job. For example, by marking '0000101-161115152703493-oozie-oozi-C'
as "BASE", users can monitor any operations they care about on these jobId.
+
+```
+from OOZIE_AUDIT_LOG_STREAM_SANDBOX[sensitivityType=='BASE' and cmd=='kill')] select * group
by user insert into oozie_audit_log_enriched_stream_out
+```
+The example policy monitors the 'kill' operation on jobId under "BASE". 
+
+# Questions on this application
+
+---
+
 
 # JMX Monitoring
 

http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_audit_log.png
----------------------------------------------------------------------
diff --git a/docs/docs/include/images/oozie_audit_log.png b/docs/docs/include/images/oozie_audit_log.png
new file mode 100644
index 0000000..42d38f6
Binary files /dev/null and b/docs/docs/include/images/oozie_audit_log.png differ

http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_install_1.png
----------------------------------------------------------------------
diff --git a/docs/docs/include/images/oozie_install_1.png b/docs/docs/include/images/oozie_install_1.png
new file mode 100644
index 0000000..238f1f5
Binary files /dev/null and b/docs/docs/include/images/oozie_install_1.png differ

http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_install_2.png
----------------------------------------------------------------------
diff --git a/docs/docs/include/images/oozie_install_2.png b/docs/docs/include/images/oozie_install_2.png
new file mode 100644
index 0000000..da4c110
Binary files /dev/null and b/docs/docs/include/images/oozie_install_2.png differ

http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_install_3.png
----------------------------------------------------------------------
diff --git a/docs/docs/include/images/oozie_install_3.png b/docs/docs/include/images/oozie_install_3.png
new file mode 100644
index 0000000..c59219e
Binary files /dev/null and b/docs/docs/include/images/oozie_install_3.png differ

http://git-wip-us.apache.org/repos/asf/eagle/blob/0c52711b/docs/docs/include/images/oozie_policy_1.png
----------------------------------------------------------------------
diff --git a/docs/docs/include/images/oozie_policy_1.png b/docs/docs/include/images/oozie_policy_1.png
new file mode 100644
index 0000000..e75dfb8
Binary files /dev/null and b/docs/docs/include/images/oozie_policy_1.png differ


Mime
View raw message