eagle-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yonzhang2...@apache.org
Subject incubator-eagle git commit: EAGLE-279 Create Documentation for Sandbox Quick starter EAGLE-279 Create Documentation for Sandbox Quick starter and JMX monitoring
Date Tue, 24 May 2016 04:28:31 GMT
Repository: incubator-eagle
Updated Branches:
  refs/heads/dev 9ad4b635d -> e618d256e


EAGLE-279 Create Documentation for Sandbox Quick starter
EAGLE-279 Create Documentation for Sandbox Quick starter and JMX monitoring

Author: Hemanth, hdendukuri@apache.org
Reviewer: yonzhang, yonzhang2012@apache.org

Closes #159


Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/e618d256
Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/e618d256
Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/e618d256

Branch: refs/heads/dev
Commit: e618d256e4854602d01a6440f51d9962be7eedea
Parents: 9ad4b63
Author: yonzhang <yonzhang2012@gmail.com>
Authored: Mon May 23 21:30:33 2016 -0700
Committer: yonzhang <yonzhang2012@gmail.com>
Committed: Mon May 23 21:30:33 2016 -0700

----------------------------------------------------------------------
 eagle-docs/images/eagle-service.png           | Bin 0 -> 449283 bytes
 eagle-docs/images/hbase-superuser.png         | Bin 0 -> 147292 bytes
 eagle-docs/images/hdfs-env-conf.png           | Bin 0 -> 460829 bytes
 eagle-docs/images/hdfs-env-conf2.png          | Bin 0 -> 265131 bytes
 eagle-docs/images/hdfs-log4j-conf.png         | Bin 0 -> 359551 bytes
 eagle-docs/images/new-site.png                | Bin 0 -> 61032 bytes
 eagle-docs/images/nn-restart.png              | Bin 0 -> 79477 bytes
 eagle-docs/images/start-storm.png             | Bin 0 -> 500714 bytes
 eagle-docs/userguide/jmx-metric-monitoring.md |  77 ++++++++++++
 eagle-docs/userguide/sandbox-quick-start.md   | 140 +++++++++++++++++++++
 10 files changed, 217 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/images/eagle-service.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/eagle-service.png b/eagle-docs/images/eagle-service.png
new file mode 100644
index 0000000..c649e36
Binary files /dev/null and b/eagle-docs/images/eagle-service.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/images/hbase-superuser.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hbase-superuser.png b/eagle-docs/images/hbase-superuser.png
new file mode 100644
index 0000000..71b4a04
Binary files /dev/null and b/eagle-docs/images/hbase-superuser.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/images/hdfs-env-conf.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-env-conf.png b/eagle-docs/images/hdfs-env-conf.png
new file mode 100644
index 0000000..b1ba80e
Binary files /dev/null and b/eagle-docs/images/hdfs-env-conf.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/images/hdfs-env-conf2.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-env-conf2.png b/eagle-docs/images/hdfs-env-conf2.png
new file mode 100644
index 0000000..069acee
Binary files /dev/null and b/eagle-docs/images/hdfs-env-conf2.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/images/hdfs-log4j-conf.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/hdfs-log4j-conf.png b/eagle-docs/images/hdfs-log4j-conf.png
new file mode 100644
index 0000000..0eade68
Binary files /dev/null and b/eagle-docs/images/hdfs-log4j-conf.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/images/new-site.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/new-site.png b/eagle-docs/images/new-site.png
new file mode 100644
index 0000000..793aa18
Binary files /dev/null and b/eagle-docs/images/new-site.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/images/nn-restart.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/nn-restart.png b/eagle-docs/images/nn-restart.png
new file mode 100644
index 0000000..562641f
Binary files /dev/null and b/eagle-docs/images/nn-restart.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/images/start-storm.png
----------------------------------------------------------------------
diff --git a/eagle-docs/images/start-storm.png b/eagle-docs/images/start-storm.png
new file mode 100644
index 0000000..705b60e
Binary files /dev/null and b/eagle-docs/images/start-storm.png differ

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/userguide/jmx-metric-monitoring.md
----------------------------------------------------------------------
diff --git a/eagle-docs/userguide/jmx-metric-monitoring.md b/eagle-docs/userguide/jmx-metric-monitoring.md
new file mode 100644
index 0000000..1a31d63
--- /dev/null
+++ b/eagle-docs/userguide/jmx-metric-monitoring.md
@@ -0,0 +1,77 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+
+
+---
+layout: doc
+title:  "JMX Metric Monitoring" 
+permalink: /docs/jmx-metric-monitoring.html
+---
+-->
+
+JMX metric for hadoop namenode url [http://127.0.0.1:50070/jmx](http://127.0.0.1:50070/jmx)
can be monitored using Eagle. Follow below steps to enable this feature in Eagle.    
+
+1. Install Python script (To populate JMX metric values to Kafka message topic periodically.)
+2. Deploy "hadoopjmx" storm topology.
+3. Create new site and policy in UI
+4. Validate policy alert.
+
+<br/>
+
+
+### **Prerequisite**
+* Complete the setup from [sandbox-quick-start.md](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/userguide/sandbox-quick-start.md)

+
+<br/>
+
+
+### **Setup**
+From Hortonworks sandbox just run below setup script to Install Pyton JMX script, Create
Kafka topic, update Hbase tables and deploy "hadoopjmx" storm topology. 
+
+    $ /usr/hdp/current/eagle/examples/hadoop-metric-sandbox-starter.sh
+    $ /usr/hdp/current/eagle/examples/hadoop-metric-policy-create.sh
+
+<br/>
+
+
+### **Application Setup in UI**
+1. Login to Eagle UI [http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/)
username and password as "admin" and "secret"
+2. Click on "Admin" from top right and click "Management" button.
+3. On Admin management page add "New Site" name "hadoopJmxMetricDataSource" by clicking on
"New Site" link.
+![add superuser](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/new-site.png)
+4. Save the changes.
+5. On eagle home page you should see new tab called "METRIC", besides "DAM".
+6. Click on "JmxMetricMonitor" under "METRIC".
+ 
+You should see safeModePolicy policy.  
+
+<br/>
+
+
+### **Demo** 
+
+* First make sure that Kafka topic "nn_jmx_metric_sandbox" is populated with JMX metric data
periodically.(To make sure that python script is running)
+ 
+        $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181
--topic nn_jmx_metric_sandbox
+
+* Genrate Alert by producing alert triggering message in Kafka topic.  
+
+
+        $ /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-producer.sh --broker-list sandbox.hortonworks.com:6667
--topic nn_jmx_metric_sandbox
+        $ {"host": "localhost", "timestamp": 1457033916718, "metric": "hadoop.namenode.fsnamesystemstate.fsstate",
"component": "namenode", "site": "sandbox", "value": 1.0}
+  

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/e618d256/eagle-docs/userguide/sandbox-quick-start.md
----------------------------------------------------------------------
diff --git a/eagle-docs/userguide/sandbox-quick-start.md b/eagle-docs/userguide/sandbox-quick-start.md
new file mode 100644
index 0000000..8ad41d3
--- /dev/null
+++ b/eagle-docs/userguide/sandbox-quick-start.md
@@ -0,0 +1,140 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements.  See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License.  You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+Guide To Install Eagle Hortonworks sandbox. 
+===============================================
+* Prerequisite
+* Download + Patch + Build
+* Setup Hadoop Environment.
+* Stream HDFS audit logs into Kafka.
+* Install Eagle.
+* Demos "HDFS File System" & "Hive" monitoring
+
+### **Prerequisite**
+* To install Eagle on a sandbox you need to run a HDP sandbox image in a virtual machine
with 8GB memory recommended.
+	1. Get Virtual Box or VMware [Virtualization environment](http://hortonworks.com/products/hortonworks-sandbox/#install)
+	2. Get [Hortonworks Sandbox v 2.2.4](http://hortonworks.com/products/hortonworks-sandbox/#archive)
+* JDK 1.7  
+* NPM (On MAC OS try "brew install node") 	
+<br/>
+
+### **Download + Patch + Build**
+* Download latest Eagle source released From Apache [[Tar]](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/apache-eagle-0.3.0-incubating-src.tar.gz)
, [[MD5]](http://www-us.apache.org/dist/incubator/eagle/apache-eagle-0.3.0-incubating/apache-eagle-0.3.0-incubating-src.tar.gz.md5)

+* Build manually with [Apache Maven](https://maven.apache.org/):
+
+     >$ tar -zxvf apache-eagle-0.3.0-incubating-src.tar.gz <br/>
+     >$ cd incubator-eagle-release-0.3.0-rc3 <br/>
+     >$ curl -O https://patch-diff.githubusercontent.com/raw/apache/incubator-eagle/pull/150.patch
<br/>
+     >$ git apply 150.patch <br/>
+     >$ mvn clean package -DskipTests <br/>
+     
+* After building successfully, you will get the tarball under `eagle-assembly/target/` named
as `eagle-0.3.0-incubating-bin.tar.gz`
+<br/>
+
+### **Setup Hadoop Environment In Sandbox**
+1. Launch Ambari to manage the Hadoop environment
+   * Enable Ambari in sandbox http://127.0.0.1:8000 (Click on Enable Button)
+   * Login to Ambari UI http://127.0.0.1:8080/ with username and password as "admin"
+2. Grant root as HBase superuser via Ambari
+![add superuser](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/hbase-superuser.png)
+3. Start HBase,Storm & Kafka from Ambari. Showing Storm as an example below. 
+![Restart Services](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/start-storm.png
"Services")
+4. If the NAT network is used in a virtual machine, it's required to add port 9099 to forwarding
ports
+  ![Forwarding Port](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/eagle-service.png)
+
+<br/>
+
+### **Stream HDFS audit logs into Kafka**   
+ 
+**Note**: This section is only needed for "HDFS File System" Monitoring.
+ 
+* **Step 1**: Configure Advanced hadoop-log4j via <a href="http://localhost:8080/#/main/services/HDFS/configs"
target="_blank">Ambari UI</a>, and add below "KAFKA_HDFS_AUDIT" log4j appender to
hdfs audit logging.
+
+	log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender
+	log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log
+	log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667
+	log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer
+	log4j.appender.KAFKA_HDFS_AUDIT.Layout=org.apache.log4j.PatternLayout
+	log4j.appender.KAFKA_HDFS_AUDIT.Layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n
+	log4j.appender.KAFKA_HDFS_AUDIT.ProducerType=async
+
+    ![HDFS LOG4J Configuration](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/hdfs-log4j-conf.png
"hdfslog4jconf")
+
+* **Step 2**: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs"
target="_blank">Ambari UI</a>, and add the reference to KAFKA_HDFS_AUDIT to HADOOP_NAMENODE_OPTS.
+
+      -Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT
+
+    ![HDFS Environment Configuration](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/hdfs-env-conf.png
"hdfsenvconf")
+
+* **Step 3**: Edit Advanced hadoop-env via <a href="http://localhost:8080/#/main/services/HDFS/configs"
target="_blank">Ambari UI</a>, and append the following command to it.
+
+      export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/*
+
+    ![HDFS Environment Configuration](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/hdfs-env-conf2.png
"hdfsenvconf2")
+
+* **Step 4**: save the changes 
+
+* **Step 5**: "Restart All" Storm & Kafka from Ambari. (Similar to starting server in
pervious step except make sure to click on "Restart All")
+
+* **Step 6**: Restart name node 
+![Restart Services](https://github.com/hdendukuri/incubator-eagle/blob/master/eagle-docs/images/nn-restart.png
"Services")
+
+* **Step 7**: Check whether logs from are flowing into topic `sandbox_hdfs_audit_log`
+      
+      > /usr/hdp/2.2.4.2-2/kafka/bin/kafka-console-consumer.sh --zookeeper sandbox.hortonworks.com:2181
--topic sandbox_hdfs_audit_log      
+      
+<br/>
+
+### **Install Eagle**
+The following installation actually contains installing and setting up a sandbox site with
HdfsAuditLog & HiveQueryLog  data sources. 
+    
+    >$ scp -P 2222  /eagle-assembly/target/eagle-0.3.0-incubating-bin.tar.gz root@127.0.0.1:/root/
<br/>
+    >$ ssh root@127.0.0.1 -p 2222 <br/>
+    >$ tar -zxvf eagle-0.3.0-incubating-bin.tar.gz <br/>
+    >$ mv eagle-0.3.0-incubating eagle <br/>
+    >$ mv eagle /usr/hdp/current/ <br/>
+    >$ cd /usr/hdp/current/eagle <br/>
+    >$ examples/eagle-sandbox-starter.sh <br/>
+
+<br/>
+
+### **Demos**
+* Login to Eagle UI [http://localhost:9099/eagle-service/](http://localhost:9099/eagle-service/)
username and password as "admin" and "secret"
+* **HDFS**:
+	1. Click on menu "DAM" and select "HDFS" to view HDFS policy
+	2. You should see policy with name "viewPrivate". This Policy generates alert when any user
reads HDFS file name "private" under "tmp" folder.
+	3. In sandbox read restricted HDFS file "/tmp/private" by using command 
+	
+	   > hadoop fs -cat "/tmp/private"
+
+	From UI click on alert tab and you should see alert for the attempt to read restricted file.
 
+* **Hive**:
+	1. Click on menu "DAM" and select "Hive" to view Hive policy
+	2. You should see policy with name "queryPhoneNumber". This Policy generates alert when
hive table with sensitivity(Phone_Number) information is queried. 
+	3. In sandbox read restricted sensitive HIVE column.
+	
+       >$ su hive <br/>
+       >$ hive <br/>
+       >$ set hive.execution.engine=mr; <br/>
+       >$ use xademo; <br/>
+       >$ select a.phone_number from customer_details a, call_detail_records b where a.phone_number=b.phone_number;
<br/>
+
+    From UI click on alert tab and you should see alert for your attempt to dfsf read restricted
column.  
+
+<br/>


Mime
View raw message