eagle-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hdenduk...@apache.org
Subject svn commit: r1743289 - in /incubator/eagle/web: _config.yml development-quick-guide.md download.md hdfs-hive-monitoring.md images/docs/new-jmx-site.png images/docs/nn-restart.png jmx-metric-monitoring.md quick-start.md
Date Wed, 11 May 2016 00:52:38 GMT
Author: hdendukuri
Date: Wed May 11 00:52:38 2016
New Revision: 1743289

URL: http://svn.apache.org/viewvc?rev=1743289&view=rev
Log:
Updated Quick Starter Guide. Added Download, JMX metric Monitoring setup, development-quick-guide
and hdfs-hive-monitoring. Modified documents menu.
https://github.com/eaglemonitoring/eaglemonitoring.github.io/pull/40

Added:
    incubator/eagle/web/development-quick-guide.md   (with props)
    incubator/eagle/web/download.md   (with props)
    incubator/eagle/web/hdfs-hive-monitoring.md   (with props)
    incubator/eagle/web/images/docs/new-jmx-site.png   (with props)
    incubator/eagle/web/images/docs/nn-restart.png   (with props)
    incubator/eagle/web/jmx-metric-monitoring.md   (with props)
Modified:
    incubator/eagle/web/_config.yml
    incubator/eagle/web/quick-start.md

Modified: incubator/eagle/web/_config.yml
URL: http://svn.apache.org/viewvc/incubator/eagle/web/_config.yml?rev=1743289&r1=1743288&r2=1743289&view=diff
==============================================================================
--- incubator/eagle/web/_config.yml (original)
+++ incubator/eagle/web/_config.yml Wed May 11 00:52:38 2016
@@ -12,13 +12,14 @@ url: "http://goeagle.io" # the base host
 markdown: kramdown
 permalink: date
 
+#host: 10.254.190.228
+#port: 4000
+
 documentations:
   - category: Getting Started
     links:
       - title: Introduction
         url: /docs/index.html
-      - title: Quick Start
-        url: /docs/quick-start.html
       - title: Terminology
         url: /docs/terminology.html
       - title: Ecosystem
@@ -29,16 +30,22 @@ documentations:
         url: /docs/FAQ.html
   - category: Installation
     links:
-      - title: Deploy Eagle in Sandbox
-        url: /docs/deployment-in-sandbox.html
-      - title: Deploy Eagle in the Production
-        url: /docs/deployment-in-production.html
+      - title: Quick Starter(Eagle In Sandbox)
+        url: /docs/quick-start.html
+      - title: Download
+        url: /docs/download.html
+      - title: HDFS & Hive Activity Monitoring
+        url: /docs/hdfs-hive-monitoring.html
+      - title: JMX URL Monitoring
+        url: /docs/jmx-metric-monitoring.html
       - title: Stream Log Data into Kafka
         url: /docs/import-hdfs-auditLog.html
-      - title: Install Eagle Ambari Plugin
-        url: /docs/ambari-plugin-install.html
-      - title: Configuration
+      - title: Eagle Cluster Deployment
+        url: /docs/deployment-in-production.html
+      - title: Eagle Configuration
         url: /docs/configuration.html
+      - title: Enable LDAP
+        url: /docs/tutorial/ldap.html
   - category: Tutorial
     links: 
       - title: Site Management Tutorial
@@ -49,13 +56,17 @@ documentations:
         url: /docs/tutorial/userprofile.html
       - title: Data Classification Tutorial
         url: /docs/tutorial/classification.html
-      - title: Eagle LDAP Tutorial
-        url: /docs/tutorial/ldap.html
+  - category: How To
+    links: 
+      - title: Use Metadata API
+        url: /docs/policy-api.html
+  - category: Development Guide
+    links: 
+      - title: Development Quick Guide
+        url: /docs/development-quick-guide.html
   - category: Advanced
     links: 
       - title: Eagle Architecture Highlights
         url: /docs/architecture-highlights.html
-      - title: Policy API Design
-        url: /docs/policy-api.html
       - title: User Profile Machine Learning
         url: /docs/user-profile-ml.html

Added: incubator/eagle/web/development-quick-guide.md
URL: http://svn.apache.org/viewvc/incubator/eagle/web/development-quick-guide.md?rev=1743289&view=auto
==============================================================================
--- incubator/eagle/web/development-quick-guide.md (added)
+++ incubator/eagle/web/development-quick-guide.md Wed May 11 00:52:38 2016
@@ -0,0 +1,30 @@
+---
+layout: doc
+title:  "Development Quick Guide" 
+permalink: /docs/development-quick-guide.html
+---
+
+### **Source Code**
+
+Apache Eagle source code is version controlled using Git version control:
+[Commits Summary](https://git-wip-us.apache.org/repos/asf?p=incubator-eagle.git) 
+
+* **Source Repo**: [https://github.com/apache/incubator-eagle.git](https://github.com/apache/incubator-eagle.git)
+* **Github**: [https://github.com/apache/incubator-eagle](https://github.com/apache/incubator-eagle)
+<br/><br/>
+
+### **Issue Tracking**
+
+Track issues on the “Eagle” Project on the Apache JIRA ([browse](https://issues.apache.org/jira/browse/EAGLE)).
+<br/><br/>
+
+### **Setup Development Env**
+
+Refer to this link for how to setup dev env: [Setup Development Env](https://github.com/apache/incubator-eagle/blob/master/eagle-docs/development/eagle_development_setup.md)
+<br/><br/>
+
+### **How to Contribute**
+
+Refer to this link for how to contribute: [Contributing to Eagle](https://cwiki.apache.org/confluence/display/EAG/Contributing+to+Eagle)
+
+<br/>
\ No newline at end of file

Propchange: incubator/eagle/web/development-quick-guide.md
------------------------------------------------------------------------------
    svn:eol-style = native

Added: incubator/eagle/web/download.md
URL: http://svn.apache.org/viewvc/incubator/eagle/web/download.md?rev=1743289&view=auto
==============================================================================
--- incubator/eagle/web/download.md (added)
+++ incubator/eagle/web/download.md Wed May 11 00:52:38 2016
@@ -0,0 +1,14 @@
+---
+layout: doc
+title:  "Apache Eagle Downloads" 
+permalink: /docs/download.html
+---
+
+### **Latest Release (Source Code)** 
+
+* [Apache Eagle Incubating v0.3.0](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/)
+* [Release Notes](https://git-wip-us.apache.org/repos/asf?p=incubator-eagle.git;a=blob_plain;f=CHANGELOG.txt;hb=refs/heads/branch-0.3.0)
+* Git Tag: [eagle-0.3.0-incubating](https://github.com/apache/incubator-eagle/releases/tag/eagle-0.3.0-incubating)
+* Git Commit: [b1d0a87ac0df1cc881427faafc637c152c180872](https://github.com/apache/incubator-eagle/commit/b1d0a87ac0df1cc881427faafc637c152c180872)
+
+<br/>
\ No newline at end of file

Propchange: incubator/eagle/web/download.md
------------------------------------------------------------------------------
    svn:eol-style = native

Added: incubator/eagle/web/hdfs-hive-monitoring.md
URL: http://svn.apache.org/viewvc/incubator/eagle/web/hdfs-hive-monitoring.md?rev=1743289&view=auto
==============================================================================
--- incubator/eagle/web/hdfs-hive-monitoring.md (added)
+++ incubator/eagle/web/hdfs-hive-monitoring.md Wed May 11 00:52:38 2016
@@ -0,0 +1,85 @@
+---
+layout: doc
+title:  "HDFS & Hive Quick Start" 
+permalink: /docs/hdfs-hive-monitoring.html
+---
+
+This Guide describes the steps to install data activity monitoring of "HDFS File System"
& "Hive".
+
+* Prerequisite
+* Stream HDFS audit logs into Kafka.
+* Demos "HDFS File System" & "Hive" monitoring
+<br/><br/>
+
+
+### **Prerequisite**
+* Complete the setup from [Quick Starter(Eagle In Sandbox)](/docs/quick-start.html)	 	
+<br/><br/>
+
+
+### **Stream HDFS audit logs into Kafka**   
+ 
+  Note: This section is only needed for "HDFS File System" Monitoring. For another option
to stream HDFS audit logs into Kafka using Logstash [Click Here](/docs/import-hdfs-auditLog.html)
+ 
+* **Step 1**: Configure Advanced hadoop-log4j via <a href="http://localhost:8080/#/main/services/HDFS/configs"
target="_blank">Ambari UI</a>, by adding below "KAFKA_HDFS_AUDIT" log4j appender
to hdfs audit logging.
+
+	   log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
+	   log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log
+	   log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667
+	   log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer
+	   log4j.appender.KAFKA_HDFS_AUDIT.Layout=org.apache.log4j.PatternLayout
+	   log4j.appender.KAFKA_HDFS_AUDIT.Layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
+	   log4j.appender.KAFKA_HDFS_AUDIT.ProducerType=async
+
+    ![HDFS LOG4J Configuration](/images/docs/hdfs-log4j-conf.png "hdfslog4jconf")
+
+* **Step 2**: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs"
target="_blank">Ambari UI</a>, and add the reference to KAFKA_HDFS_AUDIT to HADOOP_NAMENODE_OPTS.
+
+      -Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
+
+    ![HDFS Environment Configuration](/images/docs/hdfs-env-conf.png "hdfsenvconf")
+
+* **Step 3**: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs"
target="_blank">Ambari UI</a>, and append the following command to it.
+
+      export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
+
+    ![HDFS Environment Configuration](/images/docs/hdfs-env-conf2.png "hdfsenvconf2")
+
+* **Step 4**: save the changes 
+
+* **Step 5**: "Restart All" Storm & Kafka from Ambari.
+
+* **Step 6**: Restart name node 
+
+![Restart Services](/images/docs/nn-restart.png "Services")
+
+* **Step 7**: Check whether logs from "/var/log/hadoop/hdfs/hdfs-audit.log" are flowing into
topic `sandbox_hdfs_audit_log`
+      
+        $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181
--topic sandbox_hdfs_audit_log      
+      
+<br/>
+
+
+### **Demos**
+* Login to Eagle UI [http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/)
using username and password as "admin" and "secret"
+* **HDFS**:
+	1. Click on menu "DAM" and select "HDFS" to view HDFS policy
+	2. You should see policy with name "viewPrivate". This Policy generates alert when any user
reads HDFS file name "private" under "tmp" folder.
+	3. In sandbox read restricted HDFS file "/tmp/private" by using command 
+	
+	   > hadoop fs -cat /tmp/private
+
+	From UI click on alert tab and you should see alert for the attempt to read restricted file.
 
+* **Hive**:
+	1. Click on menu "DAM" and select "Hive" to view Hive policy
+	2. You should see policy with name "queryPhoneNumber". This Policy generates alert when
hive table with sensitivity(Phone_Number) information is queried. 
+	3. In sandbox read restricted sensitive HIVE column. ( To learn more about data sensitivity
settings click [Data Classification Tutorial](/docs/tutorial/classification.html))
+	
+        $ su hive <br/>
+        $ hive <br/>
+        $ set hive.execution.engine=mr; <br/>
+        $ use xademo; <br/>
+        $ select a.phone_number from customer_details a, call_detail_records b where a.phone_number=b.phone_number;
<br/>
+
+    From UI click on alert tab and you should see alert for your attempt to dfsf read restricted
column.  
+<br/>

Propchange: incubator/eagle/web/hdfs-hive-monitoring.md
------------------------------------------------------------------------------
    svn:eol-style = native

Added: incubator/eagle/web/images/docs/new-jmx-site.png
URL: http://svn.apache.org/viewvc/incubator/eagle/web/images/docs/new-jmx-site.png?rev=1743289&view=auto
==============================================================================
Binary file - no diff available.

Propchange: incubator/eagle/web/images/docs/new-jmx-site.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: incubator/eagle/web/images/docs/nn-restart.png
URL: http://svn.apache.org/viewvc/incubator/eagle/web/images/docs/nn-restart.png?rev=1743289&view=auto
==============================================================================
Binary file - no diff available.

Propchange: incubator/eagle/web/images/docs/nn-restart.png
------------------------------------------------------------------------------
    svn:mime-type = application/octet-stream

Added: incubator/eagle/web/jmx-metric-monitoring.md
URL: http://svn.apache.org/viewvc/incubator/eagle/web/jmx-metric-monitoring.md?rev=1743289&view=auto
==============================================================================
--- incubator/eagle/web/jmx-metric-monitoring.md (added)
+++ incubator/eagle/web/jmx-metric-monitoring.md Wed May 11 00:52:38 2016
@@ -0,0 +1,57 @@
+---
+layout: doc
+title:  "JMX Metric Monitoring" 
+permalink: /docs/jmx-metric-monitoring.html
+---
+
+JMX metric for hadoop namenode url [http://127.0.0.1:50070/jmx](http://127.0.0.1:50070/jmx)
can be monitored using Eagle. Follow below steps to enable this feature in Eagle.    
+
+1. Install Python script (To populate JMX metric values to Kafka topic periodically.)
+2. Deploy "hadoopjmx" storm topology.
+3. Create new site and policy in UI
+4. Validate policy alert.
+
+<br/>
+
+
+### **Prerequisite**
+* Complete the setup from [Quick Starter(Eagle In Sandbox)](/docs/quick-start.html)	
+
+<br/>
+
+
+### **Setup**
+From Hortonworks sandbox just run below setup script to Install Pyton JMX script, Create
Kafka topic, update Hbase tables and deploy "hadoopjmx" storm topology. 
+
+    $ /usr/hdp/current/eagle/examples/hadoop-metric-sandbox-starter.sh
+    $ /usr/hdp/current/eagle/examples/hadoop-metric-policy-create.sh  
+
+<br/>
+
+
+### **Application Setup in UI**
+1. Login to Eagle UI [http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/)
using username and password as "admin" and "secret"
+2. Click on "Admin" from top right and click "Management" button.
+3. On Admin management page add "New Site" name "hadoopJmxMetricDataSource", by clicking
on "New Site" link.
+![add superuser](/images/docs/new-jmx-site.png)
+4. Save the changes.
+5. On eagle home page you should see new tab called "METRIC", beside "DAM".
+6. Click on "JmxMetricMonitor" under "METRIC".
+ 
+You should see policy with name "safeModePolicy".  
+
+<br/>
+
+
+### **Demo** 
+
+* First make sure that Kafka topic "nn_jmx_metric_sandbox" is populated with JMX metric data
periodically.(To make sure that python script is running)
+ 
+        $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181
--topic nn_jmx_metric_sandbox
+
+* Genrate Alert by producing alert triggering message into Kafka topic.  
+
+
+        $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667
--topic nn_jmx_metric_sandbox
+        $ {"host": "localhost", "timestamp": 1457033916718, "metric": "hadoop.namenode.fsnamesystemstate.fsstate",
"component": "namenode", "site": "sandbox", "value": 1.0}
+  

Propchange: incubator/eagle/web/jmx-metric-monitoring.md
------------------------------------------------------------------------------
    svn:eol-style = native

Modified: incubator/eagle/web/quick-start.md
URL: http://svn.apache.org/viewvc/incubator/eagle/web/quick-start.md?rev=1743289&r1=1743288&r2=1743289&view=diff
==============================================================================
--- incubator/eagle/web/quick-start.md (original)
+++ incubator/eagle/web/quick-start.md Wed May 11 00:52:38 2016
@@ -4,65 +4,63 @@ title:  "Quick Start"
 permalink: /docs/quick-start.html
 ---
 
-This is a tutorial-style guide for users to have a quick image of Eagle. The main content
are
+Guide To Install Eagle On Hortonworks sandbox. 
 
-* Downloading
-* Installation
-* Demos
-
-### Download/Build tarball
-
-* Download tarball directly from latest released [binary package](http://66.211.190.194/eagle-0.1.0.tar.gz)
-
-* Build manually by cloning latest code from [github](https://github.com/apache/incubator-eagle)
with [Apache Maven](https://maven.apache.org/):
-
-	  $ git clone git@github.com:apache/incubator-eagle.git
-	  $ cd Eagle
-	  $ mvn clean package -DskipTests
-
-	After building successfully, you will get the tarball under `eagle-assembly/target/` named
as `eagle-${version}-bin.tar.gz`
+* Prerequisite
+* Download + Patch + Build
+* Setup Hadoop Environment.
+* Install Eagle.
+* Demo
 <br/>
 
-### Installation
-The fastest way to start with Eagle is to:
-
-* [Install Eagle with Sandbox](/docs/deployment-in-sandbox.html)
-* [Install Eagle with Docker](https://issues.apache.org/jira/browse/EAGLE-3)(under development)
-
-If you want to deploy eagle in production environment, please refer to:
-
-* [Deploy Eagle in the Production](/docs/deployment-in-production.html)
+### **Prerequisite**
+* To install Eagle on a sandbox, you need to run a HDP sandbox image in a virtual machine
(8GB memory recommended).
+	1. Get Virtual Box or VMware [Virtualization environment](http://hortonworks.com/products/hortonworks-sandbox/#install)
+	2. Get [Hortonworks Sandbox v 2.2.4](http://hortonworks.com/products/hortonworks-sandbox/#archive)
+* JDK 1.7  
+* NPM (On MAC OS try "brew install node") 	
 <br/>
 
-### Demos
-
-* Define policy with Eagle web
-    * Step 1: Select the site which is monitored by the backend topologies. For example "sandbox"
-        ![](/images/docs/selectSite.png)
-    * Step 2: Create a policy
-        ![](/images/docs/hdfs-policy1.png)
+### **Download + Patch + Build**
+* Download latest Eagle source released From Apache [[Tar]](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/apache-eagle-0.3.0-incubating-src.tar.gz)
, [[MD5]](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/apache-eagle-0.3.0-incubating-src.tar.gz.md5)

+* Build manually with [Apache Maven](https://maven.apache.org/):
+
+	  $ tar -zxvf apache-eagle-0.3.0-incubating-src.tar.gz
+	  $ cd incubator-eagle-release-0.3.0-rc3  
+	  $ curl -O https://patch-diff.githubusercontent.com/raw/apache/incubator-eagle/pull/150.patch
+	  $ git apply 150.patch
+	  $ mvn clean package -DskipTests
 
-    Learn more about how to define policy, please refer to tutorial [Policy Management](/docs/tutorial/policy.html)
+	After building successfully, you will get tarball under `eagle-assembly/target/` named as
`eagle-0.3.0-incubating-bin.tar.gz`
 <br/>
 
-* Test policy and check alerting
-
-    **Example 1** (HDFSAuditLog): validate sample policy “viewPrivate” on [Eagle
web](http://localhost:9099/eagle-service) by running a HDFS command
-
-      $ hdfs dfs -cat /tmp/private
-
-    You should see an alert for policy name “viewPrivate” in [Eagle web](http://localhost:9099/eagle-service)
. Under Alerts page.
-
-    **Example 2** (HiveQueryLog): validate sample policy “queryPhoneNumber” in
[Eagle web](http://localhost:9099/eagle-service) by submitting a hive job
+### **Setup Hadoop Environment**
+1. Launch Ambari to manage the Hadoop environment
+   * Enable Ambari in sandbox [http://127.0.0.1:8000](http://127.0.0.1:8000) (Click on Enable
Button)
+   * Login to Ambari UI [http://127.0.0.1:8080](http://127.0.0.1:8080) with username and
password as "admin"
+2. Grant root as HBase superuser via Ambari
+![add superuser](/images/docs/hbase-superuser.png)
+3. Start HBase,Storm & Kafka from Ambari. Showing Storm as an example below. 
+![Restart Services](/images/docs/start-storm.png "Services")
+4. If the NAT network is used in a virtual machine, its required to add port 9099 to "Port
Forwarding"
+  ![Port Forwarding](/images/docs/eagle-service.png)
+<br/>
 
-      $ su hive
-      $ hive
-      > set hive.execution.engine=mr;
-      > use xademo;
-      > select a.phone_number from customer_details a, call_detail_records b where a.phone_number=b.phone_number;
 
-  You should see an alert for policy name “queryPhoneNumber” in [Eagle web](http://localhost:9099/eagle-service)
. Under Alerts page.
+### **Install Eagle**
+    
+     $ scp -P 2222  eagle-assembly/target/eagle-0.3.0-incubating-bin.tar.gz root@127.0.0.1:/root/
+     $ ssh root@127.0.0.1 -p 2222 (password is hadoop)
+     $ tar -zxvf eagle-0.3.0-incubating-bin.tar.gz
+     $ mv eagle-0.3.0-incubating eagle
+     $ mv eagle /usr/hdp/current/
+     $ cd /usr/hdp/current/eagle
+     $ examples/eagle-sandbox-starter.sh
 
 <br/>
 
+### **Demos**
+* Login to Eagle UI [http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/)
using username and password as "admin" and "secret"
+* [HDFS & Hive](/docs/hdfs-hive-monitoring.html)
+* [JMX Metric Monitoring](/docs/jmx-metric-monitoring.html)
 <br/>



Mime
View raw message