Return-Path: X-Original-To: apmail-eagle-commits-archive@minotaur.apache.org Delivered-To: apmail-eagle-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DB399186BD for ; Thu, 4 Feb 2016 06:27:32 +0000 (UTC) Received: (qmail 34489 invoked by uid 500); 4 Feb 2016 06:27:32 -0000 Delivered-To: apmail-eagle-commits-archive@eagle.apache.org Received: (qmail 34459 invoked by uid 500); 4 Feb 2016 06:27:32 -0000 Mailing-List: contact commits-help@eagle.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@eagle.incubator.apache.org Delivered-To: mailing list commits@eagle.incubator.apache.org Received: (qmail 34450 invoked by uid 99); 4 Feb 2016 06:27:32 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 04 Feb 2016 06:27:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 3591D1A04F1 for ; Thu, 4 Feb 2016 06:27:32 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.573 X-Spam-Level: ***** X-Spam-Status: No, score=5.573 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_BADIPHTTP=2, KAM_LAZY_DOMAIN_SECURITY=1, KAM_LINEPADDING=1.2, LONG_HEX_URI=1, NORMAL_HTTP_TO_IP=0.001, RP_MATCHES_RCVD=-0.429, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id z0nCUOvk4Vwl for ; Thu, 4 Feb 2016 06:27:19 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTP id A439524E24 for ; Thu, 4 Feb 2016 06:27:18 +0000 (UTC) Received: from svn01-us-west.apache.org (svn.apache.org [10.41.0.6]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id C3922E0428 for ; Thu, 4 Feb 2016 06:27:17 +0000 (UTC) Received: from svn01-us-west.apache.org (localhost [127.0.0.1]) by svn01-us-west.apache.org (ASF Mail Server at svn01-us-west.apache.org) with ESMTP id C27D73A0142 for ; Thu, 4 Feb 2016 06:27:17 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: svn commit: r1728410 [1/2] - in /incubator/eagle: ./ site/ site/about/ site/css/ site/data/ site/development/ site/docs/ site/docs/tutorial/ site/fonts/ site/images/ site/images/docs/ site/images/posts/ site/images/slider/ site/images/usecases/ site/js... Date: Thu, 04 Feb 2016 06:27:17 -0000 To: commits@eagle.incubator.apache.org From: hao@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20160204062717.C27D73A0142@svn01-us-west.apache.org> Author: hao Date: Thu Feb 4 06:27:16 2016 New Revision: 1728410 URL: http://svn.apache.org/viewvc?rev=1728410&view=rev Log: Add more files Added: incubator/eagle/site/build.sh (with props) incubator/eagle/site/docs/tutorial/ldap.html (with props) incubator/eagle/web/.gitignore (with props) incubator/eagle/web/CNAME (with props) incubator/eagle/web/FAQ.md (with props) incubator/eagle/web/README.md (with props) incubator/eagle/web/_config.yml (with props) incubator/eagle/web/about.md (with props) incubator/eagle/web/ambari-plugin-install.md (with props) incubator/eagle/web/architecture-highlights.md (with props) incubator/eagle/web/build.sh (with props) incubator/eagle/web/community.md (with props) incubator/eagle/web/configuration.md (with props) incubator/eagle/web/deployment-in-production.md (with props) incubator/eagle/web/deployment-in-sandbox.md (with props) incubator/eagle/web/development.md (with props) incubator/eagle/web/documentation.md (with props) incubator/eagle/web/ecosystem.md (with props) incubator/eagle/web/feed.xml (with props) incubator/eagle/web/import-hdfs-auditLog.md (with props) incubator/eagle/web/index.html (with props) incubator/eagle/web/install.md (with props) incubator/eagle/web/introduction.md (with props) incubator/eagle/web/policy-api.md (with props) incubator/eagle/web/quick-start.md (with props) incubator/eagle/web/standalone-install.md (with props) incubator/eagle/web/terminology.md (with props) incubator/eagle/web/tutorial-classfication.md (with props) incubator/eagle/web/tutorial-ldap.md (with props) incubator/eagle/web/tutorial-policy.md (with props) incubator/eagle/web/tutorial-setup.md (with props) incubator/eagle/web/tutorial-userprofile.md (with props) incubator/eagle/web/usecase.md (with props) incubator/eagle/web/user-profile-ml.md (with props) Modified: incubator/eagle/ (props changed) incubator/eagle/site/ (props changed) incubator/eagle/site/about/ (props changed) incubator/eagle/site/css/ (props changed) incubator/eagle/site/data/ (props changed) incubator/eagle/site/development/ (props changed) incubator/eagle/site/docs/ (props changed) incubator/eagle/site/docs/tutorial/ (props changed) incubator/eagle/site/feed.xml incubator/eagle/site/fonts/ (props changed) incubator/eagle/site/images/ (props changed) incubator/eagle/site/images/docs/ (props changed) incubator/eagle/site/images/posts/ (props changed) incubator/eagle/site/images/slider/ (props changed) incubator/eagle/site/images/usecases/ (props changed) incubator/eagle/site/js/ (props changed) incubator/eagle/site/post/ (props changed) incubator/eagle/site/post/2015/ (props changed) incubator/eagle/site/post/2015/10/ (props changed) incubator/eagle/site/post/2015/10/27/ (props changed) incubator/eagle/web/_includes/ (props changed) incubator/eagle/web/_layouts/ (props changed) incubator/eagle/web/_posts/ (props changed) incubator/eagle/web/_sass/ (props changed) incubator/eagle/web/css/ (props changed) incubator/eagle/web/data/ (props changed) incubator/eagle/web/fonts/ (props changed) incubator/eagle/web/images/ (props changed) incubator/eagle/web/images/docs/ (props changed) incubator/eagle/web/images/posts/ (props changed) incubator/eagle/web/images/slider/ (props changed) incubator/eagle/web/images/usecases/ (props changed) incubator/eagle/web/js/ (props changed) Propchange: incubator/eagle/ ('svn:ignore' removed) Propchange: incubator/eagle/site/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/about/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Added: incubator/eagle/site/build.sh URL: http://svn.apache.org/viewvc/incubator/eagle/site/build.sh?rev=1728410&view=auto ============================================================================== --- incubator/eagle/site/build.sh (added) +++ incubator/eagle/site/build.sh Thu Feb 4 06:27:16 2016 @@ -0,0 +1,19 @@ +#!/bin/bash + +export web_dir=$(dirname $0) + +which jekyll 1>/dev/null 2>&1 +if [ "$?" != "0" ];then + echo "jekyll not found, please install jekyll firstly according to https://jekyllrb.com/" + exit 1 +fi + +cmd="jekyll build" +echo $cmd +$cmd + +echo "cp -rf _site/ ../site" +cp -R _site/. ../site + +echo "Done." + Propchange: incubator/eagle/site/build.sh ------------------------------------------------------------------------------ svn:eol-style = native Propchange: incubator/eagle/site/build.sh ------------------------------------------------------------------------------ svn:executable = * Propchange: incubator/eagle/site/css/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/data/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/development/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/docs/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/docs/tutorial/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Added: incubator/eagle/site/docs/tutorial/ldap.html URL: http://svn.apache.org/viewvc/incubator/eagle/site/docs/tutorial/ldap.html?rev=1728410&view=auto ============================================================================== --- incubator/eagle/site/docs/tutorial/ldap.html (added) +++ incubator/eagle/site/docs/tutorial/ldap.html Thu Feb 4 06:27:16 2016 @@ -0,0 +1,266 @@ + + + + + + Eagle - Eagle LDAP Tutorial + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+
+
+
+
+ + +
+
+
+
+
+
    +
  • Slide 1
    • +
+
+
+
+
+
+
+ + + +
+
+

Secure Hadoop in Real Time

+ +
+
+
+ + +
+ Fork eagle on GitHub + +
+
+ + +
+

Eagle LDAP Tutorial

+

To enable Eagle LDAP authentication on the web, two steps are needed.

+ +

Step 1: edit configuration under lib/tomcat/webapps/eagle-service/WEB-INF/classes/ldap.properties.

+ +
ldap.server=ldap://localhost:10389
+ldap.username=uid=admin,ou=system
+ldap.password=secret
+ldap.user.searchBase=ou=Users,o=mojo
+ldap.user.searchPattern=(uid={0})
+ldap.user.groupSearchBase=ou=groups,o=mojo
+acl.adminRole=
+acl.defaultRole=ROLE_USER
+
+ +

acl.adminRole and acl.defaultRole are two custom properties for Eagle. Eagle manages admin users with groups. If you set acl.adminRole as ROLE_{EAGLE-ADMIN-GROUP-NAME}, members in this group have admin privilege. acl.defaultRole is ROLE_USER.

+ +

Step 2: edit conf/eagle-service.conf, and add springActiveProfile=”default”

+ +
eagle{
+    service{
+        storage-type="hbase"
+        hbase-zookeeper-quorum="localhost"
+        hbase-zookeeper-property-clientPort=2181
+        zookeeper-znode-parent="/hbase",
+        springActiveProfile="default"
+    }
+}
+
+ + +
+
+ +
+
+ + + +
+
+
+
+
+Apache Eagle is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. +
+
+ + + + + + +
+
+Copyright © 2015 The Apache Software Foundation, Licensed under the Apache License, Version 2.0. +
+
+Apache Eagle, Eagle, Apache, the Apache feather logo, and the Apache Incubator project logo are trademarks of The Apache Software Foundation. +
+
+
+
+
+ + + + + + + + + + + + + \ No newline at end of file Propchange: incubator/eagle/site/docs/tutorial/ldap.html ------------------------------------------------------------------------------ svn:eol-style = native Modified: incubator/eagle/site/feed.xml URL: http://svn.apache.org/viewvc/incubator/eagle/site/feed.xml?rev=1728410&r1=1728409&r2=1728410&view=diff ============================================================================== --- incubator/eagle/site/feed.xml (original) +++ incubator/eagle/site/feed.xml Thu Feb 4 06:27:16 2016 @@ -5,8 +5,8 @@ Eagle - Secure Hadoop in Real Time http://goeagle.io/ - Thu, 04 Feb 2016 14:13:35 +0800 - Thu, 04 Feb 2016 14:13:35 +0800 + Thu, 04 Feb 2016 14:19:23 +0800 + Thu, 04 Feb 2016 14:19:23 +0800 Jekyll v2.5.3 Propchange: incubator/eagle/site/fonts/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/images/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/images/docs/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/images/posts/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/images/slider/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/images/usecases/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/js/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/post/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/post/2015/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/post/2015/10/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/site/post/2015/10/27/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Added: incubator/eagle/web/.gitignore URL: http://svn.apache.org/viewvc/incubator/eagle/web/.gitignore?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/.gitignore (added) +++ incubator/eagle/web/.gitignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1,5 @@ +_site +.sass-cache +*.DS_Store +.idea/* + Propchange: incubator/eagle/web/.gitignore ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/CNAME URL: http://svn.apache.org/viewvc/incubator/eagle/web/CNAME?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/CNAME (added) +++ incubator/eagle/web/CNAME Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +goeagle.io \ No newline at end of file Propchange: incubator/eagle/web/CNAME ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/FAQ.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/FAQ.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/FAQ.md (added) +++ incubator/eagle/web/FAQ.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,36 @@ +--- +layout: doc +title: "Frequently Asked Questions" +permalink: /docs/FAQ.html +--- + +* **Q1. Not able to access storm worker log via browser** + + > Add the following line in host machine's hosts file + + 127.0.0.1 sandbox.hortonworks.com + +* **Q2. Not able to send data into kafka using kafka console producer**: + + /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic sandbox_hdfs_audit_log + + > Kafka broker are binding to host sandbox.hortonworks.com + +* **Q3. Cannot visit eagle service url http://localhost:9099 through the browser** + + > If your network is NAT, then you need to add forwarding port 9099 for eagle service, or you should check if HBase is alive. + +* **Q4: There is no data in Kafka topic after installing log4j Kafka appender** + + > There are a few reasons for this: 1) check if the log4j kafka appender is installed successfully and there is no exceptions when restarting the namenode. 2) check if Kafka is started. 3) If everything is ok, the final choice is to restart the namenode again. + + +* **Q5: There is no workers in some topologies.** + + > Storm in sandbox has only two slots (workers) to run topologies by default. If want to run more than two topologies, a user should add extra slots in the +configurations and restart Storm + + ![Add slots](/images/docs/storm-slot.png) + + + Propchange: incubator/eagle/web/FAQ.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/README.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/README.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/README.md (added) +++ incubator/eagle/web/README.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,49 @@ +# Eagle Documentation Site +Welcome to the Eagle documentation! + +## Prerequisites + +Install [jekyll](https://jekyllrb.com/) gem + + $ gem install jekyll + +Generate the site, and start a server locally: + + $ jekyll serve -w + +The `-w` option tells jekyll to watch for changes to files and regenerate the site automatically when any content changes. + +Point your browser to [http://localhost:4000](http://localhost:4000) + +By default, jekyll will generate the site in a `_site` directory. + +## Editing documentations +1. Create a markdown file and add following content in header + + --- + layout: doc + title: "Eagle Overview" + permalink: /docs/some-new-doc.html + --- + + More content here .. + +2. Register the link in documentations sidebar navigation + + documentations: + - category: "Some category" + links: + - title: Some new doc + url: /docs/some-new-doc.html + +## Publishing the Apache Website +In order to publish the website, you must have committer access to Eagle's subversion repository. + +The Eagle website is published using Apache svnpubsub. Any changes committed to subversion will be automatically published to eagle.apache.org. + +To publish changes, tell jekyll to generate the site in the publish directory of subversion, then commit the changes: + + cd docs + jekyll build -d /path/to/svn/repo/publish + cd /path/to/svn/repo/publish + svn commit Propchange: incubator/eagle/web/README.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/_config.yml URL: http://svn.apache.org/viewvc/incubator/eagle/web/_config.yml?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/_config.yml (added) +++ incubator/eagle/web/_config.yml Thu Feb 4 06:27:16 2016 @@ -0,0 +1,61 @@ +# Site settings +title: Eagle +email: cn.haochen@gmail.com +description: Eagle - Secure Hadoop in Real Time + +baseurl: "" # the subpath of your site, e.g. /blog/ +url: "http://goeagle.io" # the base hostname & protocol for your site +# twitter_username: eagle +# github_username: eagle + +# Build settings +markdown: kramdown +permalink: date + +documentations: + - category: Getting Started + links: + - title: Introduction + url: /docs/index.html + - title: Quick Start + url: /docs/quick-start.html + - title: Terminology + url: /docs/terminology.html + - title: Ecosystem + url: /docs/ecosystem.html + - title: Community + url: /docs/community.html + - title: FAQ + url: /docs/FAQ.html + - category: Installation + links: + - title: Deploy Eagle in Sandbox + url: /docs/deployment-in-sandbox.html + - title: Deploy Eagle in the Production + url: /docs/deployment-in-production.html + - title: Stream Log Data into Kafka + url: /docs/import-hdfs-auditLog.html + - title: Install Eagle Ambari Plugin + url: /docs/ambari-plugin-install.html + - title: Configuration + url: /docs/configuration.html + - category: Tutorial + links: + - title: Site Management Tutorial + url: /docs/tutorial/setup.html + - title: Eagle Policy Tutorial + url: /docs/tutorial/policy.html + - title: User Profile Tutorial + url: /docs/tutorial/userprofile.html + - title: Data Classification Tutorial + url: /docs/tutorial/classification.html + - title: Eagle LDAP Tutorial + url: /docs/tutorial/ldap.html + - category: Advanced + links: + - title: Eagle Architecture Highlights + url: /docs/architecture-highlights.html + - title: Policy API Design + url: /docs/policy-api.html + - title: User Profile Machine Learning + url: /docs/user-profile-ml.html Propchange: incubator/eagle/web/_config.yml ------------------------------------------------------------------------------ svn:eol-style = native Propchange: incubator/eagle/web/_includes/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/_layouts/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/_posts/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/_sass/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Added: incubator/eagle/web/about.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/about.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/about.md (added) +++ incubator/eagle/web/about.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,11 @@ +--- +layout: page +title: About +permalink: /about/ +--- + +This is the base Jekyll theme. You can find out more info about customizing your Jekyll theme, as well as basic Jekyll usage documentation at [jekyllrb.com](http://jekyllrb.com/) + +You can find the source code for the Jekyll new theme at: [github.com/jglovier/jekyll-new](https://github.com/jglovier/jekyll-new) + +You can find the source code for Jekyll at [github.com/jekyll/jekyll](https://github.com/jekyll/jekyll) \ No newline at end of file Propchange: incubator/eagle/web/about.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/ambari-plugin-install.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/ambari-plugin-install.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/ambari-plugin-install.md (added) +++ incubator/eagle/web/ambari-plugin-install.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,39 @@ +--- +layout: doc +title: "Install Eagle Ambari Plugin" +permalink: /docs/ambari-plugin-install.html +--- + +Assume Eagle package has been copied and exacted under /usr/hdp/current/eagle. + +> **WARNING**: the following instructions work in sandbox currently. + + +### Pre-requisites + +1. Create a Kafka topic if you have not. Here is an example command. + + $ /usr/hdp/current/kafka-broker/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hdfs_audit_log + +2. Stream HDFS log data to Kafka, and refer to [here](/docs/import-hdfs-auditLog.html) on how to do it . + +### Installation Steps + +1. Start dependent services Storm, Spark, HBase & Kafka via Ambari. + +2. Install Eagle Ambari plugin + + $ /usr/hdp/current/eagle/bin/eagle-ambari.sh install + +3. Restart [Ambari](http://127.0.0.1:8000/) click on disable and enable Ambari back. + +4. Add Eagle Service to Ambari. Click on "Add Service" on Ambari Main page + + ![AddService](/images/docs/add-service.png "AddService") + ![Eagle Services](/images/docs/eagle-service-success.png "Eagle Services") + +5. Add Policies and meta data required by running the below script. + + $ cd + $ examples/sample-sensitivity-resource-create.sh + $ examples/sample-policy-create.sh \ No newline at end of file Propchange: incubator/eagle/web/ambari-plugin-install.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/architecture-highlights.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/architecture-highlights.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/architecture-highlights.md (added) +++ incubator/eagle/web/architecture-highlights.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,29 @@ +--- +layout: doc +title: "Eagle Architecture Highlights" +permalink: /docs/architecture-highlights.html +--- + +> Eagle is an open-source Data Activity Monitoring solution for Hadoop to instantly detect access to sensitive data or malicious activities, and to take appropriate actions. + +![](/images/docs/eagle-highlights.png) + +### Data Collection and Storage + +Eagle provides programming API for extending Eagle to integrate any data source into Eagle policy evaluation framework. For example, Eagle hdfs audit monitoring collects data from Kafka which is populated from namenode log4j appender or from logstash agent. Eagle hive monitoring collects hive query logs from running job through YARN API, which is designed to be scalable and fault-tolerant. + +Eagle uses HBase as storage for storing metadata and metrics data, and also supports relational database through configuration change. + +### Data Processing + +**Stream processing API**: Eagle provides a stream processing API which is an abstraction on Apache Storm, but is also extensible to other streaming engines. This abstraction allows developers to easily assemble data transformation, filtering, external data join, etc. without being physically bound to a specific streaming platform. The Eagle streaming API also allows developers to easily integrate business logic with the Eagle policy engine. Internally, the Eagle framework compiles business logic execution DAG into program primitives of the underlying stream infrastructure—for example, Apache Storm. + +**Alerting framework**: The Eagle alerting framework includes a stream metadata API, a policy engine provider API for extensibility, and a policy partitioner interface for scalability. + +**Machine-learning module**: Eagle provides capabilities to define user activity patterns or user profiles for Hadoop users based on the user behavior in the platform. The idea is to provide anomaly detection capability without setting hard thresholds in the system. The user profiles generated by our system are modeled using machine-learning algorithms and used for detection of anomalous user activities, where users’ activity pattern differs from their pattern history. Currently Eagle uses two algorithms for anomaly detection: Eigen-Value Decomposition and Density Estimation. The algorithms read data from HDFS audit logs, slice and dice data, and generate models for each user in the system. Once models are generated, Eagle uses the Storm framework for near-real-time anomaly detection to determine if current user activities are suspicious or not with respect to their model. The block diagram below shows the current pipeline for user profile training and online detection. + +### Eagle Service +**Policy Manager**: Eagle Policy Manager provides a UI and Restful API for users to define policies. The Eagle user interface makes it easy to manage policies with a few clicks, mark or import sensitivity metadata, perform HDFS or Hive resource browsing, access alert dashboards, etc. + +**Query Service**: Eagle provides a SQL-like service API to support comprehensive computation for huge sets of data—comprehensive filtering, aggregation, histogram, sorting, top, arithmetical expression, pagination, etc. Although Eagle supports HBase for data storage as a first-class citizen, a relational database is supported as well. For HBase storage, the Eagle query framework compiles a user-provided SQL-like query into HBase native filter objects, and then executes it through the HBase coprocessor on the fly. + Propchange: incubator/eagle/web/architecture-highlights.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/build.sh URL: http://svn.apache.org/viewvc/incubator/eagle/web/build.sh?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/build.sh (added) +++ incubator/eagle/web/build.sh Thu Feb 4 06:27:16 2016 @@ -0,0 +1,19 @@ +#!/bin/bash + +export web_dir=$(dirname $0) + +which jekyll 1>/dev/null 2>&1 +if [ "$?" != "0" ];then + echo "jekyll not found, please install jekyll firstly according to https://jekyllrb.com/" + exit 1 +fi + +cmd="jekyll build" +echo $cmd +$cmd + +echo "cp -rf _site/ ../site" +cp -R _site/. ../site + +echo "Done." + Propchange: incubator/eagle/web/build.sh ------------------------------------------------------------------------------ svn:eol-style = native Propchange: incubator/eagle/web/build.sh ------------------------------------------------------------------------------ svn:executable = * Added: incubator/eagle/web/community.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/community.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/community.md (added) +++ incubator/eagle/web/community.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,55 @@ +--- +layout: doc +title: "Eagle Community" +permalink: /docs/community.html +--- +### Mailing Lists + +|Name|Email|Subscribe|Unsubscribe|Archive| +|----|-----|---------|-----------|-------| +| User Mailling List|[user@eagle.incubator.apache.org](mailto:user@eagle.incubator.apache.org)|[subscribe](mailto:user-subscribe@eagle.incubator.apache.org)|[unsubscribe](mailto:user-unsubscribe@eagle.incubator.apache.org)| [eagle-user](http://mail-archives.apache.org/mod_mbox/eagle-user/)| +| Dev Mailling List|[dev@eagle.incubator.apache.org](mailto:dev@eagle.incubator.apache.org)|[subscribe](mailto:dev-subscribe@eagle.incubator.apache.org)|[unsubscribe](mailto:dev-unsubscribe@eagle.incubator.apache.org)| [eagle-dev](http://mail-archives.apache.org/mod_mbox/eagle-dev/)| +| Issues Mailling List|[issues@eagle.incubator.apache.org](mailto:issues@eagle.incubator.apache.org)|[subscribe](mailto:issues-subscribe@eagle.incubator.apache.org)|[unsubscribe](mailto:issues-unsubscribe@eagle.incubator.apache.org)| [eagle-issues](http://mail-archives.apache.org/mod_mbox/eagle-issues/)| +| Commits Mailling List|[commits@eagle.incubator.apache.org](mailto:commits@eagle.incubator.apache.org)|[subscribe](mailto:commits-subscribe@eagle.incubator.apache.org)|[unsubscribe](mailto:commits-unsubscribe@eagle.incubator.apache.org)| [eagle-commits](http://mail-archives.apache.org/mod_mbox/eagle-commits/)| + + +### Issue Tracker + +[https://issues.apache.org/jira/browse/EAGLE](https://issues.apache.org/jira/browse/EAGLE) + +### Social Network + +* **Twitter**: [@TheApacheEagle](https://twitter.com/TheApacheEagle) +* **Wechat**: Apache_Eagle + +### Events and Meetups + +**Conferences** + +* [**IEEE BigData 2015**](http://cci.drexel.edu/bigdata/bigdata2015/): [Eagle: User Profile-based Anomaly Detection in Hadoop Clusters](http://cci.drexel.edu/bigdata/bigdata2015/APaper.html) by [Chaitali Gupta](https://github.com/chaitaligupta) and [Edward Zhang](https://github.com/yonzhang) on Oct 29 - Nov 01, 2015 at [Santa Clara, CA, USA](https://www.google.com/maps/place/Santa+Clara,+CA) +* [**China Hadoop Summit 2015**](http://www.chinahadoop.com/english): [Hadoop Eagle - Real Time Monitoring Platform](http://www.slideshare.net/haoch/hadoop-eaglev4) by [Hao Chen](https://github.com/haoch) on July 23 - 24, 2015 at [Shanghai, China](https://www.google.com/maps/place/Shanghai,+China) +* [**Hadoop Summit 2015**](http://2015.hadoopsummit.org/san-jose/): [Hadoop Eagle - Real Time Monitoring Framework for eBay Hadoop](http://www.slideshare.net/Hadoop_Summit/hadoop-eagle-real-time-monitoring) by [Edward Zhand](https://github.com/yonzhang) and [Hao Chen](https://github.com/haoch) on June 9 - 11, 2015 at [San Jose, CA, USA](https://www.google.com/maps/place/San+Jose,+CA) +
+ +### Project Contributors + +**Initial Commiters** + +* [Edward Zhang (张勇)](https://github.com/yonzhang) +* [Hao Chen（陈浩）](https://github.com/haoch) +* [Chaitali Gupta](https://github.com/chaitaligupta) +* [Libin Sun（孙立斌）](https://github.com/sunlibin) +* [Jilin Jiang（将吉麟）](https://github.com/zombiej) +* [Qingwen Zhao（赵晴雯）](https://github.com/qingwen220) +* [Hemanth Dendukuri](https://github.com/hdendukuri) +* [Arun Manoharan](https://github.com/armanoharan) +* [Adunuthula, Seshu](https://github.com/seshuad) +
+ +### Project History + +* **Oct 20th, 2015** Eagle was officially open sourced by [eBay Inc.](http://www.ebay.com). + +### Credits + +* Thanks [eBay Inc.](http://www.ebay.com) to donated this project to open source community, first announement at [eBay Techblog](http://www.ebaytechblog.com/2015/10/23/eagle-is-your-hadoop-data-secured/). Propchange: incubator/eagle/web/community.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/configuration.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/configuration.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/configuration.md (added) +++ incubator/eagle/web/configuration.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,82 @@ +--- +layout: doc +title: "Configuration" +permalink: /docs/configuration.html +--- + +Eagle requires you to create a configuration file under `conf/` directory for each topology. This page will give detailed +description of Eagle topology configuration. + +Eagle currently supports to customize configurations for three kinds of topologies: + +* HDFS Audit Log Configuration +* Hive Query Log Configuration +* User Profile Configuration + +> Note that the properties with * are some important properties, which are those related to your cluster. + + +### HDFS Audit Log Configuration +--- + + Class ||| Property Name ||| Description + -----------------||| ------------- ||| ----------- + envContextConfig ||| env ||| currently only Storm is supported. + ||| mode ||| local or cluster + ||| **topologyName*** ||| in the format {site}-{topology-name} + ||| stormConfigFile ||| a storm configuration file for overriding some Storm properties + ||| parallelismConfig ||| parallelism for both kafka consumer and alert executors +dataSourceConfig ||| **topic** ||| Kafka topic for audit log streaming, make sure it exists + ||| **zkConnection*** ||| ZooKeeper connection string, you can also specify multiple hosts in the form hostname1:port1,hostname2:port2,hostname3:port3 + |||zkConnectionTimeoutMS ||| timeout + ||| fetchSize ||| default value + ||| deserializerClass ||| default value + |||**transactionZKServers*** ||| ZooKeeper servers, you can also specify multiple hosts in the form hostname1,hostname2,hostname3 + ||| transactionZKPort ||| ZooKeeper connection port + ||| transactionZKRoot ||| ZooKeeper chroot path for Eagle + ||| consumerGroupId ||| default is eagle.hdfsaudit.consumer + ||| transactionStateUpdateMS ||| default is 2000 +alertExecutorConfigs ||| parallelism ||| default is 1 + ||| partitioner ||| default value is eagle.alert.policy.DefaultPolicyPartitioner + ||| needValidation ||| true or false +eagleProps ||| **site*** ||| site name, such as sandbox, datacenter1, datacenter2 + ||| dataSource ||| hdfsAuditLog + ||| dataJoinPollIntervalSec ||| time interval for retrieving data from HBase + ||| **mailHost*** ||| SMTP server + ||| **mailSmtpPort*** ||| SMTP server port, default is 25 + ||| mailDebug ||| true or false + ||| eagleService.host ||| tomcat server host, default is localhost + ||| eagleService.port ||| 9099 + ||| eagleService.username ||| admin + ||| eagleService.password ||| secret + dynamicConfigSource ||| enabled ||| true or false, default is true + ||| initDelayMillis ||| default is 0 + ||| delayMillis ||| default is 30000 + + +
+ +### Hive Query Log Configuration +--- + + Class ||| Property Name ||| Description + -----------------||| ------------- ||| ----------- + envContextConfig ||| same as HDF ||| + dataSourceConfig ||| flavor ||| stormrunning + ||| **zkQuorum*** ||| ZooKeeper connection string, you can also specify multiple hosts in the form hostname1:port1,hostname2:port2,hostname3:port3 + ||| **zkRoot*** ||| ZooKeeper chroot path for Eagle to store data, default is /jobrunning + ||| zkSessionTimeoutMs ||| ZooKeeper session timeout, default is 15000 + ||| zkRetryTimes ||| ZooKeeper retry times, default is 3 + ||| zkRetryInterval ||| default is 2000 + ||| **RMEndPoints*** ||| Resource manager, for example http://localhost:8088/ + ||| **HSEndPoint*** ||| History server, for example http://localhost:19888/ + ||| partitionerCls ||| eagle.job.DefaultJobPartitionerImpl + alertExecutorConfigs ||| same as HDFS ||| + eagleProps ||| same as HDFS ||| + dynamicConfigSource ||| same as HDFS ||| + +
+ +### User Profile Configuration +--- +Please refer to the HDFS part \ No newline at end of file Propchange: incubator/eagle/web/configuration.md ------------------------------------------------------------------------------ svn:eol-style = native Propchange: incubator/eagle/web/css/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/data/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Added: incubator/eagle/web/deployment-in-production.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/deployment-in-production.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/deployment-in-production.md (added) +++ incubator/eagle/web/deployment-in-production.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,133 @@ +--- +layout: doc +title: "Deploy Eagle in the Production" +permalink: /docs/deployment-in-production.html +--- + + +This page outlines the steps for deploying Eagle in the production environment. +Notice that Eagle requires you have full permissions to HDFS, HBase and Storm CLI. + + +Here's the main content of this page: + +* Pre-requisites + * Hadoop Environment + * Stream HDFS audit log data into Kafka +* Installation + * Edit Configure files + * Install metadata +* Setup a monitoring site + * Create a site with Eagle web + * Create site related configuration files for topologies + * Submit topologies +* Stop Eagle Services + + +### **Pre-requisites** + +* **Hadoop Environment** + + * HDFS: 2.6.x + * HBase: 0.98 or later + * Storm: 0.9.3 or later + * Kafka: 0.8.x or later + * Java: 1.7.x + + +* **Stream HDFS audit log data (Only for HDFSAuditLog Monitoring)** + + 1. Create a Kafka topic for importing audit log. Here is an example command to create topic sandbox_hdfs_audit_log. + + $ cd + $ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hdfs_audit_log + + 2. Populate audit log into the Kafka topic created above, and refer to [here](/docs/import-hdfs-auditLog.html) on How to do it. + + +### **Installation** + +* Step 1: Edit environment related configurations: + + * Edit `bin/eagle-env.sh` + + # TODO: make sure java version is 1.7.x + export JAVA_HOME= + + # TODO: Storm nimbus host. Default is localhost + export EAGLE_NIMBUS_HOST=localhost + + # TODO: EAGLE_SERVICE_HOST, default is `hostname -f` + export EAGLE_SERVICE_HOST=localhost + + + * Edit `conf/eagle-service.conf` + + # TODO: hbase.zookeeper.quorum in the format host1,host2,host3,... + # default is "localhost" + hbase-zookeeper-quorum="localhost" + + # TODO: hbase.zookeeper.property.clientPort + # default is 2181 + hbase-zookeeper-property-clientPort=2181 + + # TODO: hbase configuration: zookeeper.znode.parent + # default is "/hbase" + zookeeper-znode-parent="/hbase" + +* Step 2: Install metadata for policies + + $ cd + + # create HBase tables + $ bin/eagle-service-init.sh + + # start Eagle web service + $ bin/eagle-service.sh start + + # import metadata after Eagle service is successfully started + $ bin/eagle-topology-init.sh + +### **Setup a monitoring site** + +* Step 1: Login Eagle web http://${EAGLE_SERVICE_HOST}:9099/eagle-service with account `admin/secret` + ![login](/images/docs/login.png) +* Step 2: Create a site with Eagle web + (Example: create a site "Demo" with two data sources to monitor) + ![setup a site](/images/docs/new-site.png) +* Step 3: Create site related configuration files for topologies + + Please refer to [samples](https://github.com/eBay/Eagle/tree/master/eagle-assembly/src/main/conf), and create a configuration file for each chosen datasource under $EAGLE_HOME/conf/. + More descriptions are [here](/docs/configuration.html) +* Step 4: Submit topologies + + # start HDFS audilt log monitoring + $ bin/eagle-topology.sh --main eagle.security.auditlog.HdfsAuditLogProcessorMain --config conf/XXXX-hdfsAuditLog-application.conf start + + # start Hive Query Log Monitoring + $ bin/eagle-topology.sh --main eagle.security.hive.jobrunning.HiveJobRunningMonitoringMain --config conf/XXXX-hiveQueryLog-application.conf start + + # start User Profiles + $ bin/eagle-topology.sh --main eagle.security.userprofile.UserProfileDetectionMain --config conf/XXXX-userprofile-topology.conf start + +You have now successfully installed Eagle and setup a monitoring site. Next you can + +* Create more policies with Eagle web [tutorial](/docs/tutorial/policy.html) + +* Experience alerting with instructions on [Quick Starer](/docs/quick-start.html) + +* Enable resolver and classification functions of Eagle web [tutorial](/docs/tutorial/setup.html) + +* Check topologies with Storm UI + +### **Stop Services** + +* Stop eagle service + + $ bin/eagle-service.sh stop + +* Stop eagle topologies + + $ bin/eagle-topology.sh --topology {topology-name} stop + $ bin/eagle-topology.sh --topology {topology-name} stop + $ bin/eagle-topology.sh --topology {topology-name} stop Propchange: incubator/eagle/web/deployment-in-production.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/deployment-in-sandbox.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/deployment-in-sandbox.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/deployment-in-sandbox.md (added) +++ incubator/eagle/web/deployment-in-sandbox.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,113 @@ +--- +layout: doc +title: "Deploy Eagle in Sandbox" +permalink: /docs/deployment-in-sandbox.html +--- + +Here is the summary of the steps for setting up Eagle in Hortonworks sandbox: + +* Step 1: Setup sandbox image in a virtual machine +* Step 2: Setup Hadoop environment in sandbox +* Step 3: Download and extract a Eagle release to sandbox +* Step 4: Install Eagle +* Step 5: Stream HDFS audit log + + +### **Step 1: Setup sandbox image in a virtual machine** + +To install Eagle on a sandbox you need to run a HDP sandbox image in a virtual machine with 8GB memory recommended. + +1. Get Virtual Box or VMware [Virtualization environment](http://hortonworks.com/products/hortonworks-sandbox/#install) +2. Get [Hortonworks Sandbox v 2.2.4](http://hortonworks.com/products/hortonworks-sandbox/#archive) + +### **Step 2: Setup Hadoop environment in sandbox** + +1. Launch Ambari to manage the Hadoop environment + * Enable Ambari in sandbox http://127.0.0.1:8000 (Click on Enable Button) + * Login to Ambari UI http://127.0.0.1:8080/ with username and password as "admin" +2. Grant root as HBase superuser via Ambari +![add superuser](/images/docs/hbase-superuser.png) +3. Start Storm, HBase & Kafka Ambari. Showing Storm as an example below. +![Restart Services](/images/docs/start-storm.png "Services") + +### **Step 3: Download and extract a Eagle release to sandbox** + +* **Download** + + * **Option 1**: Download eagle jar from [here](http://66.211.190.194/eagle-0.1.0.tar.gz). + + * **Option 2**: Build form source code [eagle github](https://github.com/eBay/Eagle). After successful build, ‘eagle-xxx-bin.tar.gz’ will be generated under `./eagle-assembly/target` + + # installed npm is required before compiling + $ mvn clean install -DskipTests=true +* **Copy and extract the package to sandbox** + + #extract + $ tar -zxvf eagle-0.1.0-bin.tar.gz + $ mv eagle-0.1.0 /usr/hdp/current/eagle + +
+ +### **Step 4: Install Eagle in Sandbox** + +The following installation actually contains installing and setting up a sandbox site with three data sources HdfsAuditLog, HiveQueryLog, and [User Profiles](/docs/tutorial/userprofile.html) + +* **Option 1**: Install Eagle using command line + + $ cd /usr/hdp/current/eagle + $ examples/eagle-sandbox-starter.sh + +* **Option 2**: Install Eagle using [Eagle Ambari plugin](/docs/ambari-plugin-install.html) + + +### **Step 5: Stream HDFS audit log** + +To stream HDFS audit log into Kafka, the last step is to install a namenode log4j Kafka appender (another option Logstash is [here](/docs/import-hdfs-auditLog.html)). + +* **Step 1**: Configure Advanced hadoop-log4j via Ambari UI, and add below "KAFKA_HDFS_AUDIT" log4j appender to hdfs audit logging. + + log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender + log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log + log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667 + log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer + log4j.appender.KAFKA_HDFS_AUDIT.Layout=org.apache.log4j.PatternLayout + log4j.appender.KAFKA_HDFS_AUDIT.Layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n + log4j.appender.KAFKA_HDFS_AUDIT.ProducerType=async + #log4j.appender.KAFKA_HDFS_AUDIT.BatchSize=1 + #log4j.appender.KAFKA_HDFS_AUDIT.QueueSize=1 + + ![HDFS LOG4J Configuration](/images/docs/hdfs-log4j-conf.png "hdfslog4jconf") + +* **Step 3**: Edit Advanced hadoop-env via Ambari UI, and add the reference to KAFKA_HDFS_AUDIT to HADOOP_NAMENODE_OPTS. + + -Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT + + ![HDFS Environment Configuration](/images/docs/hdfs-env-conf.png "hdfsenvconf") + +* **Step 4**: Edit Advanced hadoop-env via Ambari UI, and append the following command to it. + + export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/current/eagle/lib/log4jkafka/lib/* + + ![HDFS Environment Configuration](/images/docs/hdfs-env-conf2.png "hdfsenvconf2") + +* **Step 5**: save the changes and restart the namenode. + +* **Step 6**: Check whether logs are flowing into topic `sandbox_hdfs_audit_log` + + $ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log + +Now please login to Eagle web http://localhost:9099/eagle-service with account `admin/secret`, and try the sample demos on +[Quick Starer](/docs/quick-start.html) + + + (If the NAT network is used in a virtual machine, it's required to add port 9099 to forwarding ports) + ![Forwarding Port](/images/docs/eagle-service.png) + ![login](/images/docs/login.png) + + + + + + + + Propchange: incubator/eagle/web/deployment-in-sandbox.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/development.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/development.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/development.md (added) +++ incubator/eagle/web/development.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,14 @@ +--- +layout: dev +title: "Eagle Development" +permalink: /development/index.html +--- + +Start to write from here with [`markdown`](http://daringfireball.net/projects/markdown/) + +## Sample Content + +How to develop Eagle. + +how to make it? +[https://jekyllrb.com/docs/templates/](https://jekyllrb.com/docs/templates/) \ No newline at end of file Propchange: incubator/eagle/web/development.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/documentation.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/documentation.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/documentation.md (added) +++ incubator/eagle/web/documentation.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,41 @@ +--- +layout: doc +title: "Eagle Overview" +permalink: /docs/index.html +--- + +## Some example + + +#### Eagle Web Configuration + + +>- Sample about syntax highlight + +{% highlight batch %} +# eagle configuration +eagle{ + # eagle web service configuration + service{ + # storage type: ["hbase","jdbc"] + # default is "hbase" + storage-type="hbase" + + # hbase configuration: hbase.zookeeper.quorum + # default is "localhost" + hbase-zookeeper-quorum="localhost" + + # hbase configuration: hbase.zookeeper.property.clientPort + # default is 2181 + hbase-zookeeper-property-clientPort=2181 + + # hbase configuration: zookeeper.znode.parent + # default is "/hbase" + zookeeper-znode-parent="/hbase-unsecure" + } +} + +{% endhighlight %} + +how to make it? +[https://jekyllrb.com/docs/templates/](https://jekyllrb.com/docs/templates/) \ No newline at end of file Propchange: incubator/eagle/web/documentation.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/ecosystem.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/ecosystem.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/ecosystem.md (added) +++ incubator/eagle/web/ecosystem.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,19 @@ +--- +layout: doc +title: "Ecosystem" +permalink: /docs/ecosystem.html +--- + +### Eagle Core + +The core of eagle is the distributed real-time policy framework, including highly abstracted Streaming Programing API, CEP-based Policy Execution Engine, SQL-Like REST API based on HBase, Alert Deduplication, Alert Notification Framework and so on. + +### Extension + +Eagle supports pluggable policy execution engined like siddhi, machine learning and more, could support any types of data sources. + +### Integration +Eagle provides Ambari plugin for easily installation, can be easily intergrated into existing hadoop cluster. + +### User Interface +Eagle provide highly user-friendly UI for policy management \ No newline at end of file Propchange: incubator/eagle/web/ecosystem.md ------------------------------------------------------------------------------ svn:eol-style = native Added: incubator/eagle/web/feed.xml URL: http://svn.apache.org/viewvc/incubator/eagle/web/feed.xml?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/feed.xml (added) +++ incubator/eagle/web/feed.xml Thu Feb 4 06:27:16 2016 @@ -0,0 +1,30 @@ +--- +layout: null +--- + + + + {{ site.title | xml_escape }} + {{ site.description | xml_escape }} + {{ site.url }}{{ site.baseurl }}/ + + {{ site.time | date_to_rfc822 }} + {{ site.time | date_to_rfc822 }} + Jekyll v{{ jekyll.version }} + {% for post in site.posts limit:10 %} + + {{ post.title | xml_escape }} + {{ post.content | xml_escape }} + {{ post.date | date_to_rfc822 }} + {{ post.url | prepend: site.baseurl | prepend: site.url }} + {{ post.url | prepend: site.baseurl | prepend: site.url }} + {% for tag in post.tags %} + {{ tag | xml_escape }} + {% endfor %} + {% for cat in post.categories %} + {{ cat | xml_escape }} + {% endfor %} + + {% endfor %} + + Propchange: incubator/eagle/web/feed.xml ------------------------------------------------------------------------------ svn:eol-style = native Propchange: incubator/eagle/web/fonts/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/images/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/images/docs/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/images/posts/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/images/slider/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Propchange: incubator/eagle/web/images/usecases/ ------------------------------------------------------------------------------ --- svn:ignore (added) +++ svn:ignore Thu Feb 4 06:27:16 2016 @@ -0,0 +1 @@ +.* Added: incubator/eagle/web/import-hdfs-auditLog.md URL: http://svn.apache.org/viewvc/incubator/eagle/web/import-hdfs-auditLog.md?rev=1728410&view=auto ============================================================================== --- incubator/eagle/web/import-hdfs-auditLog.md (added) +++ incubator/eagle/web/import-hdfs-auditLog.md Thu Feb 4 06:27:16 2016 @@ -0,0 +1,126 @@ +--- +layout: doc +title: "How to stream hdfs log data into Kafka" +permalink: /docs/import-hdfs-auditLog.html +--- + +As Eagle consumes the data via Kafka topics in some topologies, such as HDFS audit log. To enable the full function of monitoring, a user needs to stream its data into a Kafka topic. + +There are two ways to do that. The first one is **Logstash**, which naturally supports Kafka as the output plugin; the second one is to +install a **namenode log4j Kafka appender**. + +### Logstash-kafka + +* **Step 1**: Create a Kafka topic as the streaming input. + + Here is an sample Kafka command to create topic 'sandbox_hdfs_audit_log' + + cd + bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hdfs_audit_log + +* **Step 2**: Install Logstash-kafka plugin + + * For Logstash 1.5.x, logstash-kafka has been intergrated into [logstash-input-kafka](https://github.com/logstash-plugins/logstash-input-kafka) and [logstash-output-kafka](https://github.com/logstash-plugins/logstash-output-kafka), + and released with the 1.5 version of Logstash. So you can directly use it. + + * For Logstash 1.4.x, a user should install [logstash-kafka](https://github.com/joekiller/logstash-kafka) firstly. Notice that this version **does not support partition\_key\_format**. + +* **Step 3**: Create a Logstash configuration file under ${LOGSTASH_HOME}/conf. Here is a sample. + + input { + file { + type => "hdp-nn-audit" + path => "/path/to/audit.log" + start_position => end + sincedb_path => "/var/log/logstash/" + } + } + + filter{ + if [type] == "hdp-nn-audit" { + grok { + match => ["message", "ugi=(?([\w\d\-]+))@|ugi=(?([\w\d\-]+))/[\w\d\-.]+@|ugi=(?([\w\d.\-_]+))[\s(]+"] + } + } + } + + output { + if [type] == "hdp-nn-audit" { + kafka { + codec => plain { + format => "%{message}" + } + broker_list => "localhost:9092" + topic_id => "sandbox_hdfs_audit_log" + request_required_acks => 0 + request_timeout_ms => 10000 + producer_type => "async" + message_send_max_retries => 3 + retry_backoff_ms => 100 + queue_buffering_max_ms => 5000 + queue_enqueue_timeout_ms => 5000 + batch_num_messages => 200 + send_buffer_bytes => 102400 + client_id => "hdp-nn-audit" + partition_key_format => "%{user}" + } + # stdout { codec => rubydebug } + } + } + +* **Step 4**: Start Logstash + + bin/logstash -f conf/sample.conf + +* **Step 5**: Check whether logs are flowing into the kafka topic specified by `topic_id` + +### Log4j Kafka Appender + +> Notice that if you use Ambari, such as in sandbox, you **must** follow below steps via Ambari UI. In addition, restarting namenode is required. + +* **Step 1**: Create a Kafka topic. Here is a example Kafka command for creating topic "sandbox_hdfs_audit_log" + + cd + bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic sandbox_hdfs_audit_log + +* **Step 2**: Configure $HADOOP_CONF_DIR/log4j.properties, and add a log4j appender "KAFKA_HDFS_AUDIT" to hdfs audit logging + + log4j.appender.KAFKA_HDFS_AUDIT=org.apache.eagle.log4j.kafka.KafkaLog4jAppender + log4j.appender.KAFKA_HDFS_AUDIT.Topic=sandbox_hdfs_audit_log + log4j.appender.KAFKA_HDFS_AUDIT.BrokerList=sandbox.hortonworks.com:6667 + log4j.appender.KAFKA_HDFS_AUDIT.KeyClass=org.apache.eagle.log4j.kafka.hadoop.AuditLogKeyer + log4j.appender.KAFKA_HDFS_AUDIT.Layout=org.apache.log4j.PatternLayout + log4j.appender.KAFKA_HDFS_AUDIT.Layout.ConversionPattern=%d{ISO8601} %p %c{2}: %m%n + log4j.appender.KAFKA_HDFS_AUDIT.ProducerType=async + #log4j.appender.KAFKA_HDFS_AUDIT.BatchSize=1 + #log4j.appender.KAFKA_HDFS_AUDIT.QueueSize=1 + + ![HDFS LOG4J Configuration](/images/docs/hdfs-log4j-conf.png "hdfslog4jconf") + +* **Step 3**: Edit $HADOOP_CONF_DIR/hadoop-env.sh, and add the reference to KAFKA_HDFS_AUDIT to HADOOP_NAMENODE_OPTS. + + -Dhdfs.audit.logger=INFO,DRFAAUDIT,KAFKA_HDFS_AUDIT + + ![HDFS Environment Configuration](/images/docs/hdfs-env-conf.png "hdfsenvconf") + +* **Step 4**: Edit $HADOOP_CONF_DIR/hadoop-env.sh, and append the following command to it. + + export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/path/to/eagle/lib/log4jkafka/lib/* + + ![HDFS Environment Configuration](/images/docs/hdfs-env-conf2.png "hdfsenvconf2") + +* **Step 5**: save the changes and restart the namenode. + +* **Step 6**: Check whether logs are flowing into Topic sandbox_hdfs_audit_log + + $ /usr/hdp/current/kafka-broker/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic sandbox_hdfs_audit_log + + + + + + + + + + Propchange: incubator/eagle/web/import-hdfs-auditLog.md ------------------------------------------------------------------------------ svn:eol-style = native