eagle-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From m.@apache.org
Subject [20/47] incubator-eagle git commit: EAGLE-280 Mapr integration EAGLE-280 EAGLE-282 EAGLE-284 EAGLE-307: Mapr integration
Date Mon, 25 Jul 2016 09:36:57 GMT
EAGLE-280 Mapr integration
EAGLE-280 EAGLE-282 EAGLE-284 EAGLE-307: Mapr integration

Author: Daniel Zhou, DadanielZ
Reviewer: Yong Zhang, yonzhang2012@gmail.com

Closes #201


Project: http://git-wip-us.apache.org/repos/asf/incubator-eagle/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-eagle/commit/253f99bc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-eagle/tree/253f99bc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-eagle/diff/253f99bc

Branch: refs/heads/master
Commit: 253f99bce6fb72fd18823dca0d496516fda6dde0
Parents: 5578cfd
Author: yonzhang <yonzhang2012@gmail.com>
Authored: Wed May 25 22:13:02 2016 -0700
Committer: yonzhang <yonzhang2012@gmail.com>
Committed: Wed May 25 22:13:02 2016 -0700

----------------------------------------------------------------------
 .../src/main/docs/logstash-kafka-conf.md        |  59 ++++-
 .../eagle-alert-notification-plugin/pom.xml     |   4 -
 .../org/apache/eagle/common/DateTimeUtil.java   |  12 ++
 .../security/hdfs/MAPRFSAuditLogObject.java     |  31 +++
 .../security/hdfs/MAPRFSAuditLogParser.java     |  67 ++++++
 .../crawler/audit/TestMAPRFSAuditLogParser.java |  64 ++++++
 .../eagle-security-maprfs-auditlog/pom.xml      |  74 +++++++
 .../MapRFSAuditLogKafkaDeserializer.java        |  69 ++++++
 .../auditlog/MapRFSAuditLogProcessorMain.java   | 115 ++++++++++
 .../src/main/resources/application.conf         |  69 ++++++
 .../src/main/resources/log4j.properties         |  40 ++++
 .../src/main/resources/maprFSAuditLog-init.sh   | 213 +++++++++++++++++++
 .../main/resources/security-auditlog-storm.yaml |  18 ++
 .../eagle-security-maprfs-web/pom.xml           |  70 ++++++
 .../hdfs/resolver/MAPRFSCommandResolver.java    |  72 +++++++
 .../hdfs/resolver/MAPRStatusCodeResolver.java   |  81 +++++++
 eagle-security/pom.xml                          |   2 +
 eagle-webservice/pom.xml                        |   5 +
 pom.xml                                         | 203 +++++++++++++-----
 19 files changed, 1205 insertions(+), 63 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-assembly/src/main/docs/logstash-kafka-conf.md
----------------------------------------------------------------------
diff --git a/eagle-assembly/src/main/docs/logstash-kafka-conf.md b/eagle-assembly/src/main/docs/logstash-kafka-conf.md
index 84952e0..9003fb4 100644
--- a/eagle-assembly/src/main/docs/logstash-kafka-conf.md
+++ b/eagle-assembly/src/main/docs/logstash-kafka-conf.md
@@ -22,7 +22,7 @@
 ### Install logstash-kafka plugin
 
 
-#### For Logstash 1.5.x
+#### For Logstash 1.5.x, 2.x
 
 
 logstash-kafka has been intergrated into [logstash-input-kafka][logstash-input-kafka] and [logstash-output-kafka][logstash-output-kafka], you can directly use it.
@@ -32,7 +32,7 @@ logstash-kafka has been intergrated into [logstash-input-kafka][logstash-input-k
 
 #### For Logstash 1.4.x
 
-In logstash 1.4.x, the online version does not support specifying partition\_key for Kafka producer, and data will be produced into each partitions in turn. For eagle, we need to use the src in hdfs\_audit\_log as the partition key, so some hacking work have been done. If you have the same requirment, you can follow it. 
+In logstash 1.4.x, the online version does not support specifying partition\_key for Kafka producer, and data will be produced into each partitions in turn. For eagle, we need to use the src in hdfs\_audit\_log as the partition key, so some hacking work have been done. If you have the same requirment, you can follow it.
 
 1. Install logstash-kafka
 
@@ -51,13 +51,13 @@ In logstash 1.4.x, the online version does not support specifying partition\_key
 2. Hacking the kafka.rb
 
    We have added partition\_key\_format, which is used to specify the partition_key and supported by logstash 1.5.x, into  lib/logstash/outputs/kafka.rb. More details are shown [here](https://github.xyz.com/eagle/eagle/blob/master/eagle-assembly/src/main/docs/kafka.rb).
-       
+
           config :partition_key_format, :validate => :string, :default => nil
-        
+
           public
           def register
             LogStash::Logger.setup_log4j(@logger)
-        
+
             options = {
                 :broker_list => @broker_list,
                 :compression_codec => @compression_codec,
@@ -80,9 +80,9 @@ In logstash 1.4.x, the online version does not support specifying partition\_key
             }
             @producer = Kafka::Producer.new(options)
             @producer.connect
-        
+
             @logger.info('Registering kafka producer', :topic_id => @topic_id, :broker_list => @broker_list)
-        
+
             @codec.on_event do |data|
               begin
                 @producer.send_msg(@current_topic_id,@partition_key,data)
@@ -94,7 +94,7 @@ In logstash 1.4.x, the online version does not support specifying partition\_key
               end
             end
           end # def register
-        
+
           def receive(event)
             return unless output?(event)
             if event == LogStash::SHUTDOWN
@@ -111,6 +111,9 @@ In logstash 1.4.x, the online version does not support specifying partition\_key
 
 ### Create logstash configuration file
 Go to the logstash root dir, and create a configure file
+The 2.0 release of Logstash includes a new version of the Kafka output plugin with significant configuration changes. For more details, please check the documentation pages for the [Logstash1.5](https://www.elastic.co/guide/en/logstash/1.5/plugins-outputs-kafka.html) and [Logstash2.0](https://www.elastic.co/guide/en/logstash/2.0/plugins-outputs-kafka.html) version of the kafka output plugin.
+
+#### For Logstash 1.4.X, 1.5.X
 
         input {
             file {
@@ -153,6 +156,46 @@ Go to the logstash root dir, and create a configure file
             }
         }
 
+#### For Logstash 2.X
+
+		input {
+			file {
+				type => "hdp-nn-audit"
+				path => "/path/to/audit.log"
+				start_position => end
+				sincedb_path => "/var/log/logstash/"
+			}
+		}
+
+
+		filter{
+			if [type] == "hdp-nn-audit" {
+			  grok {
+				  match => ["message", "ugi=(?<user>([\w\d\-]+))@|ugi=(?<user>([\w\d\-]+))/[\w\d\-.]+@|ugi=(?<user>([\w\d.\-_]+))[\s(]+"]
+			  }
+			}
+		}
+
+		output {
+			 if [type] == "hdp-nn-audit" {
+				  kafka {
+					  codec => plain {
+						  format => "%{message}"
+					  }
+					  bootstrap_servers => "localhost:9092"
+					  topic_id => "hdfs_audit_log"
+					  acks => “0”
+					  timeout_ms => 10000
+					  retries => 3
+					  retry_backoff_ms => 100
+					  batch_size => 16384
+					  send_buffer_bytes => 131072
+					  client_id => "hdp-nn-audit"
+				  }
+				  # stdout { codec => rubydebug }
+			  }
+		}
+
 #### grok pattern testing
 We have 3 typical patterns for ugi field as follows
 2015-02-11 15:00:00,000 INFO FSNamesystem.audit: allowed=true	ugi=user1@xyz.com (auth:TOKEN)	ip=/10.115.44.55	cmd=open	src=/apps/hdmi-technology/b_pulsar_coe/schema/avroschema/Session.avsc	dst=null	perm=null

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml b/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml
index 005af4e..504abb8 100644
--- a/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml
+++ b/eagle-core/eagle-alert/eagle-alert-notification-plugin/pom.xml
@@ -80,9 +80,5 @@
               </exclusion>
           </exclusions>
       </dependency>
-      <dependency>
-          <groupId>org.apache.hadoop</groupId>
-          <artifactId>hadoop-hdfs</artifactId>
-      </dependency>
   </dependencies>
 </project>

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java
----------------------------------------------------------------------
diff --git a/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java b/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java
index dd3a4c6..0f5b189 100644
--- a/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java
+++ b/eagle-core/eagle-query/eagle-common/src/main/java/org/apache/eagle/common/DateTimeUtil.java
@@ -100,6 +100,18 @@ public class DateTimeUtil {
 			return 0L;
 		}
 	}
+
+	//For mapr
+	//exp: 2015-06-06T10:44:22.800Z
+	public static long maprhumanDateToMilliseconds(String date) throws ParseException{
+		date = date.replace('T',' ');
+		date = date.replace('Z',' ');
+		date = date.replace('.',',');
+		SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss,SSS ");
+		sdf.setTimeZone(CURRENT_TIME_ZONE);
+		Date d = sdf.parse(date);
+		return d.getTime();
+	}
 	/**
 	 * this could be accurate only when timezone is UTC
 	 * for the timezones other than UTC, there is possibly issue, for example

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogObject.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogObject.java b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogObject.java
new file mode 100644
index 0000000..630687f
--- /dev/null
+++ b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogObject.java
@@ -0,0 +1,31 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *    http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+package org.apache.eagle.security.hdfs;
+
+
+public class MAPRFSAuditLogObject {
+    public long timestamp;
+    public String host;
+    public String status;
+    public String user;
+    public String cmd;
+    public String src;
+    public String dst;
+    public String volume;
+}

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogParser.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogParser.java b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogParser.java
new file mode 100644
index 0000000..458de12
--- /dev/null
+++ b/eagle-security/eagle-security-common/src/main/java/org/apache/eagle/security/hdfs/MAPRFSAuditLogParser.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.security.hdfs;
+
+import org.apache.eagle.common.DateTimeUtil;
+import org.codehaus.jettison.json.JSONException;
+import org.codehaus.jettison.json.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.text.ParseException;
+
+
+public final class MAPRFSAuditLogParser {
+    private final static Logger LOG = LoggerFactory.getLogger(MAPRFSAuditLogParser.class);
+
+    public MAPRFSAuditLogParser(){
+    }
+
+    public MAPRFSAuditLogObject parse(String log) throws JSONException, ParseException {
+        JSONObject jsonObject = new JSONObject(log);
+        String timestamp = jsonObject.getJSONObject("timestamp").getString("$date");
+        String cmd = jsonObject.getString("operation");
+        String user = jsonObject.getString("uid");
+        String ip = jsonObject.getString("ipAddress");
+        String status = jsonObject.getString("status");
+        String volumeID = jsonObject.getString("volumeId");
+        String src;
+        String dst;
+        if(jsonObject.has("srcFid")){
+            src = jsonObject.getString("srcFid");
+        }else{
+            src = "null";
+        }
+
+        if(jsonObject.has("dstFid")){
+            dst = jsonObject.getString("dstFid");
+        }else{
+            dst = "null";
+        }
+
+        MAPRFSAuditLogObject entity = new MAPRFSAuditLogObject();
+        entity.user = user;
+        entity.cmd = cmd;
+        entity.src = src;
+        entity.dst = dst;
+        entity.host = ip;
+        entity.status = status;
+        entity.volume = volumeID;
+        entity.timestamp = DateTimeUtil.maprhumanDateToMilliseconds(timestamp);
+        return entity;
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-common/src/test/java/org/apache/eagle/security/crawler/audit/TestMAPRFSAuditLogParser.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-common/src/test/java/org/apache/eagle/security/crawler/audit/TestMAPRFSAuditLogParser.java b/eagle-security/eagle-security-common/src/test/java/org/apache/eagle/security/crawler/audit/TestMAPRFSAuditLogParser.java
new file mode 100644
index 0000000..f0eaf10
--- /dev/null
+++ b/eagle-security/eagle-security-common/src/test/java/org/apache/eagle/security/crawler/audit/TestMAPRFSAuditLogParser.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.security.crawler.audit;
+
+import junit.framework.Assert;
+import org.apache.eagle.security.hdfs.HDFSAuditLogParser;
+import org.apache.eagle.security.hdfs.MAPRFSAuditLogObject;
+import org.apache.eagle.security.hdfs.MAPRFSAuditLogParser;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+public class TestMAPRFSAuditLogParser {
+
+    private final static Logger LOG = LoggerFactory.getLogger(TestMAPRFSAuditLogParser.class);
+    @Test
+    public void testMAPRParser() {
+        String log = "{\"timestamp\":{\"$date\":\"2015-06-06T10:44:22.800Z\"},\"operation\":\"MKDIR\",\"uid\":0,\"ipAddress\":\n" +
+                "\"10.10.104.51\",\"parentFid\":\"2049.51.131248\",\"childFid\":\"2049.56.131258\",\"childName\":\n" +
+                "\"ycsbTmp_1433587462796\",\"volumeId\":68048396,\"status\":0}";
+        MAPRFSAuditLogParser parser = new MAPRFSAuditLogParser();
+        MAPRFSAuditLogObject entity = new MAPRFSAuditLogObject();
+        try {
+            entity = parser.parse(log);
+        }catch (Exception e){
+            LOG.info(e.getMessage());
+        }
+
+        Assert.assertEquals("MKDIR",entity.cmd);
+        Assert.assertEquals("0",entity.user);
+        Assert.assertEquals("10.10.104.51",entity.host);
+        Assert.assertEquals("null",entity.src);
+        Assert.assertEquals("null",entity.dst);
+        Assert.assertEquals("0", entity.status);
+
+        log = "{\"timestamp\":{\"$date\":\"2016-02-19T01:50:01.962Z\"},\"operation\":\"LOOKUP\",\"uid\":5000,\"ipAddress\":\"192.168.6.148\",\"srcFid\":\"2049.40.131192\",\"dstFid\":\"2049.1032.133268\",\"srcName\":\"share\",\"volumeId\":186635570,\"status\":0}\n";
+        try {
+            entity = parser.parse(log);
+        }catch (Exception e){
+            LOG.info(e.getMessage());
+        }
+        Assert.assertEquals("LOOKUP",entity.cmd);
+        Assert.assertEquals("5000",entity.user);
+        Assert.assertEquals("192.168.6.148",entity.host);
+        Assert.assertEquals("2049.40.131192",entity.src);
+        Assert.assertEquals("2049.1032.133268",entity.dst);
+        Assert.assertEquals("0", entity.status);
+    }
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/pom.xml b/eagle-security/eagle-security-maprfs-auditlog/pom.xml
new file mode 100644
index 0000000..71d9127
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/pom.xml
@@ -0,0 +1,74 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~    http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <parent>
+        <groupId>org.apache.eagle</groupId>
+        <artifactId>eagle-security-parent</artifactId>
+        <version>0.4.0-incubating-SNAPSHOT</version>
+    </parent>
+    <artifactId>eagle-security-maprfs-auditlog</artifactId>
+    <packaging>jar</packaging>
+    <name>eagle-security-maprfs-auditlog</name>
+    <url>http://maven.apache.org</url>
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.eagle</groupId>
+            <artifactId>eagle-security-common</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.eagle</groupId>
+            <artifactId>eagle-stream-application-manager</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.eagle</groupId>
+            <artifactId>eagle-security-hdfs-auditlog</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.eagle</groupId>
+            <artifactId>eagle-embed-server</artifactId>
+            <version>${project.version}</version>
+            <classifier>tests</classifier>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.eagle</groupId>
+            <artifactId>eagle-embed-server</artifactId>
+            <version>${project.version}</version>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.eagle</groupId>
+            <artifactId>eagle-embed-hbase</artifactId>
+            <version>${project.version}</version>
+            <classifier>tests</classifier>
+            <scope>test</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.eagle</groupId>
+            <artifactId>eagle-embed-hbase</artifactId>
+            <version>${project.version}</version>
+            <scope>test</scope>
+        </dependency>
+    </dependencies>
+</project>

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogKafkaDeserializer.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogKafkaDeserializer.java b/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogKafkaDeserializer.java
new file mode 100644
index 0000000..5eb18cd
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogKafkaDeserializer.java
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.security.auditlog;
+
+import org.apache.eagle.dataproc.impl.storm.kafka.SpoutKafkaMessageDeserializer;
+import org.apache.eagle.security.hdfs.MAPRFSAuditLogObject;
+import org.apache.eagle.security.hdfs.MAPRFSAuditLogParser;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+import java.util.Properties;
+import java.util.TreeMap;
+
+public class MapRFSAuditLogKafkaDeserializer implements SpoutKafkaMessageDeserializer {
+    private static Logger LOG = LoggerFactory.getLogger(MapRFSAuditLogKafkaDeserializer.class);
+    private Properties props;
+
+    public MapRFSAuditLogKafkaDeserializer(Properties props){
+        this.props = props;
+    }
+
+    /**
+     * the steps for deserializing message from kafka
+     * 1. convert byte array to string
+     * 2. parse string to eagle entity
+     */
+    @Override
+    public Object deserialize(byte[] arg0) {
+        String logLine = new String(arg0);
+
+        MAPRFSAuditLogParser parser = new MAPRFSAuditLogParser();
+        MAPRFSAuditLogObject entity = null;
+        try{
+            entity = parser.parse(logLine);
+        }catch(Exception ex){
+            LOG.error("Failing parse audit log message", ex);
+        }
+        if(entity == null){
+            LOG.warn("Event ignored as it can't be correctly parsed, the log is ", logLine);
+            return null;
+        }
+        Map<String, Object> map = new TreeMap<String, Object>();
+        map.put("src", entity.src);
+        map.put("dst", entity.dst);
+        map.put("host", entity.host);
+        map.put("timestamp", entity.timestamp);
+        map.put("status", entity.status);
+        map.put("user", entity.user);
+        map.put("cmd", entity.cmd);
+        map.put("volume", entity.volume);
+
+        return map;
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogProcessorMain.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogProcessorMain.java b/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogProcessorMain.java
new file mode 100644
index 0000000..b824e8a
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/java/org/apache/eagle/security/auditlog/MapRFSAuditLogProcessorMain.java
@@ -0,0 +1,115 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *    http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+package org.apache.eagle.security.auditlog;
+
+import backtype.storm.spout.SchemeAsMultiScheme;
+import com.typesafe.config.Config;
+import org.apache.commons.lang3.time.DateUtils;
+import org.apache.eagle.common.config.EagleConfigConstants;
+import org.apache.eagle.dataproc.impl.storm.kafka.KafkaSourcedSpoutProvider;
+import org.apache.eagle.dataproc.impl.storm.kafka.KafkaSourcedSpoutScheme;
+import org.apache.eagle.datastream.ExecutionEnvironments;
+import org.apache.eagle.datastream.core.StreamProducer;
+import org.apache.eagle.datastream.storm.StormExecutionEnvironment;
+import org.apache.eagle.partition.DataDistributionDao;
+import org.apache.eagle.partition.PartitionAlgorithm;
+import org.apache.eagle.partition.PartitionStrategy;
+import org.apache.eagle.partition.PartitionStrategyImpl;
+import org.apache.eagle.security.partition.DataDistributionDaoImpl;
+import org.apache.eagle.security.partition.GreedyPartitionAlgorithm;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+public class MapRFSAuditLogProcessorMain {
+
+    public static PartitionStrategy createStrategy(Config config) {
+        // TODO: Refactor configuration structure to avoid repeated config processing configure ~ hao
+        String host = config.getString(EagleConfigConstants.EAGLE_PROPS + "." + EagleConfigConstants.EAGLE_SERVICE + "." + EagleConfigConstants.HOST);
+        Integer port = config.getInt(EagleConfigConstants.EAGLE_PROPS + "." + EagleConfigConstants.EAGLE_SERVICE + "." + EagleConfigConstants.PORT);
+        String username = config.getString(EagleConfigConstants.EAGLE_PROPS + "." + EagleConfigConstants.EAGLE_SERVICE + "." + EagleConfigConstants.USERNAME);
+        String password = config.getString(EagleConfigConstants.EAGLE_PROPS + "." + EagleConfigConstants.EAGLE_SERVICE + "." + EagleConfigConstants.PASSWORD);
+        String topic = config.getString("dataSourceConfig.topic");
+        DataDistributionDao dao = new DataDistributionDaoImpl(host, port, username, password, topic);
+        PartitionAlgorithm algorithm = new GreedyPartitionAlgorithm();
+        String key1 = EagleConfigConstants.EAGLE_PROPS + ".partitionRefreshIntervalInMin";
+        Integer partitionRefreshIntervalInMin = config.hasPath(key1) ? config.getInt(key1) : 60;
+        String key2 = EagleConfigConstants.EAGLE_PROPS + ".kafkaStatisticRangeInMin";
+        Integer kafkaStatisticRangeInMin =  config.hasPath(key2) ? config.getInt(key2) : 60;
+        PartitionStrategy strategy = new PartitionStrategyImpl(dao, algorithm, partitionRefreshIntervalInMin * DateUtils.MILLIS_PER_MINUTE, kafkaStatisticRangeInMin * DateUtils.MILLIS_PER_MINUTE);
+        return strategy;
+    }
+
+    public static KafkaSourcedSpoutProvider createProvider(Config config) {
+         String deserClsName = config.getString("dataSourceConfig.deserializerClass");
+         final KafkaSourcedSpoutScheme scheme = new KafkaSourcedSpoutScheme(deserClsName, config) {
+                 @Override
+                 public List<Object> deserialize(byte[] ser) {
+                         Object tmp = deserializer.deserialize(ser);
+                         Map<String, Object> map = (Map<String, Object>)tmp;
+                         if(tmp == null) return null;
+                         return Arrays.asList(map.get("user"), tmp);
+                 }
+         };
+
+         KafkaSourcedSpoutProvider provider = new KafkaSourcedSpoutProvider() {
+                 @Override
+                 public SchemeAsMultiScheme getStreamScheme(String deserClsName, Config context) {
+                         return new SchemeAsMultiScheme(scheme);
+                  }
+         };
+         return provider;
+    }
+
+    @SuppressWarnings("unchecked")
+    public static void execWithDefaultPartition(StormExecutionEnvironment env, KafkaSourcedSpoutProvider provider) {
+        StreamProducer source = env.fromSpout(provider).withOutputFields(2).nameAs("kafkaMsgConsumer").groupBy(Arrays.asList(0));
+        //StreamProducer reassembler = source.flatMap(new HdfsUserCommandReassembler()).groupBy(Arrays.asList(0));
+        //source.streamUnion(reassembler)
+        source.flatMap(new FileSensitivityDataJoinExecutor()).groupBy(Arrays.asList(0))
+              .flatMap(new IPZoneDataJoinExecutor())
+              .alertWithConsumer("maprFSAuditLogEventStream", "maprFSAuditLogAlertExecutor");
+        env.execute();
+    }
+
+    @SuppressWarnings("unchecked")
+    public static void execWithBalancedPartition(StormExecutionEnvironment env, KafkaSourcedSpoutProvider provider) {
+        PartitionStrategy strategy = createStrategy(env.getConfig());
+        StreamProducer source = env.fromSpout(provider).withOutputFields(2).nameAs("kafkaMsgConsumer").groupBy(strategy);
+        //StreamProducer reassembler = source.flatMap(new HdfsUserCommandReassembler()).groupBy(Arrays.asList(0));
+        //source.streamUnion(reassembler)
+        source.flatMap(new FileSensitivityDataJoinExecutor()).groupBy(Arrays.asList(0))
+                .flatMap(new IPZoneDataJoinExecutor())
+                .alertWithConsumer("maprFSAuditLogEventStream", "maprFSAuditLogAlertExecutor");
+        env.execute();
+    }
+
+	public static void main(String[] args) throws Exception{
+        StormExecutionEnvironment env = ExecutionEnvironments.getStorm(args);
+        Config config = env.getConfig();
+        KafkaSourcedSpoutProvider provider = createProvider(env.getConfig());
+        Boolean balancePartition = config.hasPath("eagleProps.balancePartitionEnabled") && config.getBoolean("eagleProps.balancePartitionEnabled");
+        if (balancePartition) {
+            execWithBalancedPartition(env, provider);
+        } else {
+            execWithDefaultPartition(env, provider);
+        }
+	}
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/application.conf
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/application.conf b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/application.conf
new file mode 100644
index 0000000..52a31a5
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/application.conf
@@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{
+  "envContextConfig" : {
+    "env" : "storm",
+    "mode" : "local",
+    "topologyName" : "maprFSAuditLogProcessTopology",
+    "stormConfigFile" : "security-auditlog-storm.yaml",
+    "parallelismConfig" : {
+      "kafkaMsgConsumer" : 1,
+      "maprFSAuditLogAlertExecutor*" : 1
+    }
+  },
+  "dataSourceConfig": {
+    "topic" : "maprtest",
+    "zkConnection" : "192.168.6.148:5181",
+    "zkConnectionTimeoutMS" : 15000,
+    "consumerGroupId" : "EagleConsumer",
+    "fetchSize" : 1048586,
+    "deserializerClass" : "org.apache.eagle.security.auditlog.MapRFSAuditLogKafkaDeserializer",
+    "transactionZKServers" : "192.168.6.148",
+    "transactionZKPort" : 5181,
+    "transactionZKRoot" : "/consumers",
+    "transactionStateUpdateMS" : 2000
+  },
+  "alertExecutorConfigs" : {
+     "maprFSAuditLogAlertExecutor" : {
+       "parallelism" : 1,
+       "partitioner" : "org.apache.eagle.policy.DefaultPolicyPartitioner"
+       "needValidation" : "true"
+     }
+  },
+  "eagleProps" : {
+    "site" : "sandbox",
+    "application": "maprFSAuditLog",
+  	"dataJoinPollIntervalSec" : 30,
+    "mailHost" : "mailHost.com",
+    "mailSmtpPort":"25",
+    "mailDebug" : "true",
+    "balancePartitionEnabled" : false,
+    #"partitionRefreshIntervalInMin" : 60,
+    #"kafkaStatisticRangeInMin" : 60,
+    "eagleService": {
+      "host": "localhost",
+      "port": 9099,
+      "username": "admin",
+      "password": "secret"
+    },
+    "readHdfsUserCommandPatternFrom" : "file"
+  },
+  "dynamicConfigSource" : {
+  	"enabled" : true,
+  	"initDelayMillis" : 0,
+  	"delayMillis" : 30000
+  }
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/log4j.properties
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/log4j.properties b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/log4j.properties
new file mode 100644
index 0000000..07f8402
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/log4j.properties
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+log4j.rootLogger=INFO, stdout, DRFA
+
+eagle.log.dir=./logs
+eagle.log.file=eagle.log
+
+
+#log4j.logger.org.apache.eagle.security.auditlog.IPZoneDataJoinExecutor=DEBUG
+#log4j.logger.org.apache.eagle.security.auditlog.FileSensitivityDataJoinExecutor=DEBUG
+log4j.logger.org.apache.eagle.security.auditlog.HdfsUserCommandReassembler=DEBUG
+#log4j.logger.org.apache.eagle.executor.AlertExecutor=DEBUG
+# standard output
+log4j.appender.stdout=org.apache.log4j.ConsoleAppender
+log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
+log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} %p [%t] %c{2}[%L]: %m%n
+
+# Daily Rolling File Appender
+log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
+log4j.appender.DRFA.File=${eagle.log.dir}/${eagle.log.file}
+log4j.appender.DRFA.DatePattern=yyyy-MM-dd
+## 30-day backup
+# log4j.appender.DRFA.MaxBackupIndex=30
+log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
+
+# Pattern format: Date LogLevel LoggerName LogMessage
+log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p [%t] %c{2}[%L]: %m%n
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh
new file mode 100644
index 0000000..5f99190
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/maprFSAuditLog-init.sh
@@ -0,0 +1,213 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with`
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# EAGLE_SERVICE_HOST, default is `hostname -f`
+export EAGLE_SERVICE_HOST=localhost
+# EAGLE_SERVICE_PORT, default is 9099
+export EAGLE_SERVICE_PORT=9099
+# EAGLE_SERVICE_USER
+export EAGLE_SERVICE_USER=admin
+# EAGLE_SERVICE_PASSWORD
+export EAGLE_SERVICE_PASSWD=secret
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+ "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=SiteApplicationService" \
+  -d '
+  [
+     {
+        "tags":{
+           "site":"sandbox",
+           "application":"maprFSAuditLog"
+        },
+        "enabled": true,
+        "config": "classification.fs.defaultFS=maprfs:///"
+     }
+  ]
+  '
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+ "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=ApplicationDescService" \
+  -d '
+  [
+     {
+        "tags":{
+           "application":"maprFSAuditLog"
+        },
+        "description":"mapr AuditLog Monitoring",
+        "alias":"MapRFSAuditLogMonitor",
+        "groupName":"MapR",
+        "features":["common","metadata", "classification"],
+	"config":"{\n\t\"view\": {\n\t\t\"prefix\": \"fileSensitivity\",\n\t\t\"service\": \"FileSensitivityService\",\n\t\t\"keys\": [\n\t\t\t\"filedir\",\n\t\t\t\"sensitivityType\"\n\t\t],\n\t\t\"type\": \"folder\",\n\t\t\"api\": \"hdfsResource\"\n\t}\n}"
+     }
+  ]
+  '
+
+
+
+## AlertStreamService
+echo ""
+echo "Importing AlertStreamService for MapRFS... "
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+ "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertStreamService" \
+ -d '
+ [
+    {
+       "tags":{
+          "application":"maprFSAuditLog",
+          "streamName":"maprFSAuditLogEventStream"
+       },
+       "description":"mapr fs audit log data source stream"
+    }
+ ]
+ '
+## AlertExecutorService: what alert streams are consumed by alert executor
+echo ""
+echo "Importing AlertExecutorService for MapRFS... "
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+ "http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertExecutorService" \
+ -d '
+ [
+    {
+       "tags":{
+          "application":"maprFSAuditLog",
+          "alertExecutorId":"maprFSAuditLogAlertExecutor",
+          "streamName":"maprFSAuditLogEventStream"
+       },
+       "description":"executor for mapr fs audit log stream"
+    }
+ ]
+ '
+## AlertStreamSchemaService: schema for event from alert stream
+echo ""
+echo "Importing AlertStreamSchemaService for MapRFS... "
+curl -u ${EAGLE_SERVICE_USER}:${EAGLE_SERVICE_PASSWD} -X POST -H 'Content-Type:application/json' \
+"http://${EAGLE_SERVICE_HOST}:${EAGLE_SERVICE_PORT}/eagle-service/rest/entities?serviceName=AlertStreamSchemaService" \
+ -d '
+ [
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "status"
+       },
+       "attrDescription": "the status of action result,  numeric codes other than 0 for success can appear, for more info, check: http://doc.mapr.com/display/MapR/Status+Codes+That+Can+Appear+in+Audit+Logs",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": "org.apache.eagle.service.security.hdfs.resolver.MAPRStatusCodeResolver"
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "cmd"
+       },
+       "attrDescription": "file/directory operation, such as MKDIR, LOOKUP",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": "org.apache.eagle.service.security.hdfs.resolver.MAPRFSCommandResolver"
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "volume"
+       },
+       "attrDescription": "volume name in mapr",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": ""
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "dst"
+       },
+       "attrDescription": "destination file or directory, such as /tmp",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": "org.apache.eagle.service.security.hdfs.resolver.HDFSResourceResolver"
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "src"
+       },
+       "attrDescription": "source file or directory, such as /tmp",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": "HDFSResourceResolver"
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "host"
+       },
+       "attrDescription": "hostname, such as localhost",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": ""
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "sensitivityType"
+       },
+       "attrDescription": "sensitivity type of of target file/folder, eg: mark such as AUDITLOG, SECURITYLOG",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": "org.apache.eagle.service.security.hdfs.resolver.HDFSSensitivityTypeResolver"
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "securityZone"
+       },
+       "attrDescription": "",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": ""
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "user"
+       },
+       "attrDescription": "process user",
+       "attrType": "string",
+       "category": "",
+       "attrValueResolver": ""
+    },
+    {
+       "tags": {
+          "application": "maprFSAuditLog",
+          "streamName": "maprFSAuditLogEventStream",
+          "attrName": "timestamp"
+       },
+       "attrDescription": "milliseconds of the datetime",
+       "attrType": "long",
+       "category": "",
+       "attrValueResolver": ""
+    }
+
+ ]
+ '
+## Finished
+echo ""
+echo "Finished initialization for eagle topology"

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/security-auditlog-storm.yaml
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/security-auditlog-storm.yaml b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/security-auditlog-storm.yaml
new file mode 100644
index 0000000..a68a323
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-auditlog/src/main/resources/security-auditlog-storm.yaml
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+topology.workers: 1
+topology.acker.executors: 1
+topology.tasks: 1
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-web/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-web/pom.xml b/eagle-security/eagle-security-maprfs-web/pom.xml
new file mode 100644
index 0000000..d10d21a
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-web/pom.xml
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~    http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+
+<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+  <modelVersion>4.0.0</modelVersion>
+  <parent>
+    <groupId>org.apache.eagle</groupId>
+    <artifactId>eagle-security-parent</artifactId>
+    <version>0.4.0-incubating-SNAPSHOT</version>
+  </parent>
+
+  <artifactId>eagle-security-maprfs-web</artifactId>
+  <name>eagle-security-maprfs-web</name>
+  <url>http://maven.apache.org</url>
+  <dependencies>
+      <dependency>
+          <groupId>org.apache.eagle</groupId>
+          <artifactId>eagle-security-hdfs-web</artifactId>
+          <version>${project.version}</version>
+      </dependency>
+      <dependency>
+          <groupId>org.apache.eagle</groupId>
+          <artifactId>eagle-entity-base</artifactId>
+          <version>${project.version}</version>
+          <exclusions>
+              <exclusion>
+                  <groupId>org.slf4j</groupId>
+                  <artifactId>log4j-over-slf4j</artifactId>
+              </exclusion>
+          </exclusions>
+      </dependency>
+      <dependency>
+          <groupId>com.sun.jersey</groupId>
+          <artifactId>jersey-server</artifactId>
+      </dependency>
+      <dependency>
+          <groupId>org.apache.eagle</groupId>
+          <artifactId>eagle-service-base</artifactId>
+          <version>${project.version}</version>
+          <exclusions>
+              <exclusion>
+                  <groupId>org.slf4j</groupId>
+                  <artifactId>log4j-over-slf4j</artifactId>
+              </exclusion>
+          </exclusions>
+      </dependency>
+      <dependency>
+          <groupId>org.apache.eagle</groupId>
+          <artifactId>eagle-security-common</artifactId>
+          <version>${project.version}</version>
+      </dependency>
+  </dependencies>
+</project>
+

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRFSCommandResolver.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRFSCommandResolver.java b/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRFSCommandResolver.java
new file mode 100644
index 0000000..edb0737
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRFSCommandResolver.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.eagle.service.security.hdfs.resolver;
+
+import org.apache.eagle.service.alert.resolver.AttributeResolvable;
+import org.apache.eagle.service.alert.resolver.AttributeResolveException;
+import org.apache.eagle.service.alert.resolver.BadAttributeResolveRequestException;
+import org.apache.eagle.service.alert.resolver.GenericAttributeResolveRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Pattern;
+
+
+public class MAPRFSCommandResolver implements AttributeResolvable<GenericAttributeResolveRequest,String> {
+    private final static Logger LOG = LoggerFactory.getLogger(MAPRFSCommandResolver.class);
+
+    private final static String [] cmdStrs = {"CHGRP", "CHOWN", "CHPERM","CREATE", "DELETE",
+            "DISABLEAUDIT", "ENABLEAUDIT", "GETATTR","LOOKUP", "MKDIR", "READ", "READDIR",
+            "RENAME", "RMDIR", "SETATTR", "WRITE"};
+
+    private final static String MAPRFS_CMD_RESOLVE_FORMAT_HINT = String.format("maprfs command must be in {%s}", StringUtils.join(cmdStrs, ","));
+
+    private final static List<String> commands = Arrays.asList(cmdStrs);
+
+    public List<String> resolve(GenericAttributeResolveRequest request) throws AttributeResolveException {
+        String query = request.getQuery().trim();
+        List<String> res = new ArrayList<>();
+        for(String cmd : commands) {
+            Pattern pattern = Pattern.compile("^" + query, Pattern.CASE_INSENSITIVE);
+            if(pattern.matcher(cmd).find()) {
+                res.add(cmd);
+            }
+        }
+        if(res.size() == 0) {
+            return commands;
+        }
+        return res;
+    }
+
+    @Override
+    public void validateRequest(GenericAttributeResolveRequest request) throws BadAttributeResolveRequestException {
+        String query = request.getQuery();
+        boolean matched = Pattern.matches("[a-zA-Z]+", query);
+        if (query == null || !matched) {
+            throw new BadAttributeResolveRequestException(MAPRFS_CMD_RESOLVE_FORMAT_HINT);
+        }
+    }
+
+    @Override
+    public Class<GenericAttributeResolveRequest> getRequestClass() {
+        return GenericAttributeResolveRequest.class;
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRStatusCodeResolver.java
----------------------------------------------------------------------
diff --git a/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRStatusCodeResolver.java b/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRStatusCodeResolver.java
new file mode 100644
index 0000000..248c297
--- /dev/null
+++ b/eagle-security/eagle-security-maprfs-web/src/main/java/org/apache/eagle/service/security/hdfs/resolver/MAPRStatusCodeResolver.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.eagle.service.security.hdfs.resolver;
+
+import org.apache.eagle.service.alert.resolver.AttributeResolvable;
+import org.apache.eagle.service.alert.resolver.AttributeResolveException;
+import org.apache.eagle.service.alert.resolver.BadAttributeResolveRequestException;
+import org.apache.eagle.service.alert.resolver.GenericAttributeResolveRequest;
+import org.apache.commons.lang3.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.regex.Pattern;
+
+
+public class MAPRStatusCodeResolver implements AttributeResolvable<GenericAttributeResolveRequest,String> {
+    private final static Logger LOG = LoggerFactory.getLogger(MAPRStatusCodeResolver.class);
+
+    private final static String [] statusCodes = {"SUCCESS","EPERM","ENOENT","EINTR","EIO","ENXIO","E2BIG","ENOEXEC","EBADF",
+            "ECHILD","EAGAIN","ENOMEM","EACCES","EFAULT","ENOTBLK","EBUSY","EEXIST","EXDEV","ENODEV","ENOTDIR","EISDIR","EINVAL",
+            "ENFILE","EMFILE","ENOTTY","ETXTBSY","EFBIG","ENOSPC","ESPIPE","EROFS","EMLINK","EPIPE","EDOM","ERANGE","EDEADLK","ENAMETOOLONG",
+            "ENOLCK","ENOSYS","ENOTEMPTY","ELOOP","EWOULDBLOCK","ENOMSG","EIDRM","ECHRNG","EL2NSYNC","EL3HLT","EL3RST","ELNRNG","EUNATCH",
+            "ENOCSI","EL2HLT","EBADE","EBADR","EXFULL","ENOANO","EBADRQC","EBADSLT","EDEADLOCK","EBFONT","ENOSTR","ENODATA","ETIME","ENOSR",
+            "ENONET","ENOPKG","EREMOTE","ENOLINK","EADV","ESRMNT","ECOMM","EPROTO","EMULTIHOP","EDOTDOT","EBADMSG","EOVERFLOW","ENOTUNIQ",
+            "EBADFD","EREMCHG","ELIBACC","ELIBBAD","ELIBSCN","ELIBMAX","ELIBEXEC","EILSEQ","ERESTART","ESTRPIPE","EUSERS","ENOTSOCK",
+            "EDESTADDRREQ","EMSGSIZE","EPROTOTYPE","ENOPROTOOPT","EPROTONOSUPPORT","ESOCKTNOSUPPORT","EOPNOTSUPP","EPFNOSUPPORT","EAFNOSUPPORT",
+            "EADDRINUSE","EADDRNOTAVAIL","ENETDOWN","ENETUNREACH","ENETRESET","ECONNABORTED","ECONNRESET","ENOBUFS","EISCONN","ENOTCONN",
+            "ESHUTDOWN","ETOOMANYREFS","ETIMEDOUT","ECONNREFUSED","EHOSTDOWN","EHOSTUNREACH","EALREADY","EINPROGRESS","ESTALE","EUCLEAN","ENOTNAM",
+            "ENAVAIL","EISNAM","EREMOTEIO","EDQUOT","ENOMEDIUM","EMEDIUMTYPE","ECANCELED","ENOKEY","EKEYEXPIRED","EKEYREVOKED","EKEYREJECTED"};
+
+    private final static String MAPRFS_STATUS_CODE_RESOLVE_FORMAT_HINT = String.format("Status code must be in {%s}", StringUtils.join(statusCodes, ","));
+
+    private final static List<String> statusList = Arrays.asList(statusCodes);
+
+    public List<String> resolve(GenericAttributeResolveRequest request) throws AttributeResolveException {
+        String query = request.getQuery().trim();
+        List<String> res = new ArrayList<>();
+        for(String status : statusList) {
+            Pattern pattern = Pattern.compile("^" + query, Pattern.CASE_INSENSITIVE);
+            if(pattern.matcher(status).find()) {
+                res.add(status);
+            }
+        }
+        if(res.size() == 0) {
+            return statusList;
+        }
+        return res;
+    }
+
+    @Override
+    public void validateRequest(GenericAttributeResolveRequest request) throws BadAttributeResolveRequestException {
+        String query = request.getQuery();
+        boolean matched = Pattern.matches("[a-zA-Z]+", query);
+        if (query == null || !matched) {
+            throw new BadAttributeResolveRequestException(MAPRFS_STATUS_CODE_RESOLVE_FORMAT_HINT);
+        }
+    }
+
+    @Override
+    public Class<GenericAttributeResolveRequest> getRequestClass() {
+        return GenericAttributeResolveRequest.class;
+    }
+}

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-security/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-security/pom.xml b/eagle-security/pom.xml
index ce1a178..93c5aa1 100644
--- a/eagle-security/pom.xml
+++ b/eagle-security/pom.xml
@@ -34,10 +34,12 @@
   <modules>
     <module>eagle-security-common</module>
 	<module>eagle-security-hdfs-auditlog</module>
+    <module>eagle-security-maprfs-auditlog</module>
     <module>eagle-security-userprofile</module>
     <module>eagle-security-hive</module>
     <module>eagle-security-hive-web</module>
     <module>eagle-security-hdfs-web</module>
+    <module>eagle-security-maprfs-web</module>
     <module>eagle-security-hdfs-securitylog</module>
     <module>eagle-security-hbase-securitylog</module>
     <module>eagle-security-hbase-web</module>

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/eagle-webservice/pom.xml
----------------------------------------------------------------------
diff --git a/eagle-webservice/pom.xml b/eagle-webservice/pom.xml
index 44bd4da..c316db2 100644
--- a/eagle-webservice/pom.xml
+++ b/eagle-webservice/pom.xml
@@ -145,6 +145,11 @@
 			<artifactId>eagle-security-hdfs-web</artifactId>
 			<version>${project.version}</version>
 		</dependency>
+		<dependency>
+			<groupId>org.apache.eagle</groupId>
+			<artifactId>eagle-security-maprfs-web</artifactId>
+			<version>${project.version}</version>
+		</dependency>
 		<!--
 		<dependency>
 			<groupId>org.apache.eagle</groupId>

http://git-wip-us.apache.org/repos/asf/incubator-eagle/blob/253f99bc/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 79d5e9d..f17e082 100755
--- a/pom.xml
+++ b/pom.xml
@@ -164,11 +164,16 @@
         <hadoop.version>2.6.0.2.2.5.1-3</hadoop.version>
         <hbase.version>0.98.4.2.2.5.1-3-hadoop2</hbase.version>
         <hive.version>1.2.1</hive.version>
+
+        <mapr-hive.version>1.2.0-mapr-1510</mapr-hive.version>
+        <mapr-hadoop.version>2.7.0-mapr-1506</mapr-hadoop.version>
+        <maprfs.version>5.0.0-mapr</maprfs.version>
+        <mapr-hbase.version>0.98.9-mapr-1503</mapr-hbase.version>
+
         <spark.core.version>1.4.0</spark.core.version>
 
         <!--  Client -->
         <kafka-client.version>0.9.0.0</kafka-client.version>
-
         <!-- Reflection -->
         <reflections.version>0.9.8</reflections.version>
 
@@ -215,6 +220,12 @@
         <jgrapht.version>0.9.0</jgrapht.version>
         <storm-kafka.version>0.9.3.2.2.0.0-2041</storm-kafka.version>
         <storm.version>0.9.3</storm.version>
+
+        <mapr-kafka.version>0.8.1.1</mapr-kafka.version>
+        <mapr-kafka-clients.version>0.8.3-mapr-1509</mapr-kafka-clients.version>
+        <mapr-storm-kafka.version>0.9.3</mapr-storm-kafka.version>
+        <mapr-storm.version>0.9.3-mapr-1509</mapr-storm.version>
+
         <curator.version>2.8.0</curator.version>
 
         <!-- Query -->
@@ -239,7 +250,7 @@
         <jaxb-impl.version>2.2.6</jaxb-impl.version>
         <stax-api.version>1.0.1</stax-api.version>
         <org.mortbay.jetty.version>6.1.26</org.mortbay.jetty.version>
-                
+
         <!-- servlet  -->
         <servlet-api.version>2.5</servlet-api.version>
 
@@ -475,11 +486,6 @@
             </dependency>
             <dependency>
                 <groupId>org.apache.kafka</groupId>
-                <artifactId>kafka_${scala.version}</artifactId>
-                <version>${kafka.version}</version>
-            </dependency>
-            <dependency>
-                <groupId>org.apache.kafka</groupId>
                 <artifactId>kafka-clients</artifactId>
                 <version>${kafka-clients.version}</version>
             </dependency>
@@ -503,25 +509,6 @@
                 </exclusions>
             </dependency>
             <dependency>
-                <groupId>org.apache.storm</groupId>
-                <artifactId>storm-kafka</artifactId>
-                <version>${storm-kafka.version}</version>
-                <exclusions>
-                    <exclusion>
-                        <groupId>ch.qos.logback</groupId>
-                        <artifactId>logback-classic</artifactId>
-                    </exclusion>
-                    <exclusion>
-                        <groupId>log4j</groupId>
-                        <artifactId>log4j</artifactId>
-                    </exclusion>
-                    <exclusion>
-                        <groupId>org.slf4j</groupId>
-                        <artifactId>log4j-over-slf4j</artifactId>
-                    </exclusion>
-                </exclusions>
-            </dependency>
-            <dependency>
                 <groupId>org.apache.spark</groupId>
                 <artifactId>spark-core_${scala.version}</artifactId>
                 <version>${spark.core.version}</version>
@@ -751,6 +738,27 @@
             <activation>
                 <activeByDefault>true</activeByDefault>
             </activation>
+            <repositories>
+                <repository>
+                    <id>HDP Release Repository</id>
+                    <url>http://repo.hortonworks.com/content/repositories/releases/</url>
+                    <releases>
+                        <enabled>true</enabled>
+                        <updatePolicy>never</updatePolicy>
+                    </releases>
+                </repository>
+                <repository>
+                    <id>HDP Central Repository</id>
+                    <url>http://repo.hortonworks.com/content/repositories/central/</url>
+                    <releases>
+                        <enabled>true</enabled>
+                        <updatePolicy>never</updatePolicy>
+                        <checksumPolicy>warn</checksumPolicy>
+                    </releases>
+                    <name>HDP Central</name>
+                    <layout>default</layout>
+                </repository>
+            </repositories>
             <dependencyManagement>
                 <dependencies>
                     <dependency>
@@ -758,7 +766,6 @@
                         <artifactId>hadoop-common</artifactId>
                         <version>${hadoop.version}</version>
                     </dependency>
-
                     <dependency>
                         <groupId>org.apache.hbase</groupId>
                         <artifactId>hbase-client</artifactId>
@@ -799,6 +806,119 @@
                         <artifactId>hive-jdbc</artifactId>
                         <version>${hive.version}</version>
                     </dependency>
+                    <dependency>
+                        <groupId>org.apache.kafka</groupId>
+                        <artifactId>kafka_${scala.version}</artifactId>
+                        <version>${kafka.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.storm</groupId>
+                        <artifactId>storm-kafka</artifactId>
+                        <version>${storm-kafka.version}</version>
+                        <exclusions>
+                            <exclusion>
+                                <groupId>ch.qos.logback</groupId>
+                                <artifactId>logback-classic</artifactId>
+                            </exclusion>
+                            <exclusion>
+                                <groupId>log4j</groupId>
+                                <artifactId>log4j</artifactId>
+                            </exclusion>
+                            <exclusion>
+                                <groupId>org.slf4j</groupId>
+                                <artifactId>log4j-over-slf4j</artifactId>
+                            </exclusion>
+                        </exclusions>
+                    </dependency>
+                </dependencies>
+            </dependencyManagement>
+        </profile>
+
+        <profile>
+            <id>mapr</id>
+            <!--<activation>-->
+                <!--<activeByDefault>true</activeByDefault>-->
+            <!--</activation>-->
+            <repositories>
+                <repository>
+                    <id>mapr-releases</id>
+                    <url>http://repository.mapr.com/maven/</url>
+                    <snapshots><enabled>false</enabled></snapshots>
+                    <releases><enabled>true</enabled></releases>
+                </repository>
+            </repositories>
+            <dependencyManagement>
+                <dependencies>
+                    <dependency>
+                        <groupId>org.apache.hadoop</groupId>
+                        <artifactId>hadoop-common</artifactId>
+                        <version>${mapr-hadoop.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.hbase</groupId>
+                        <artifactId>hbase-client</artifactId>
+                        <version>${mapr-hbase.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>com.mapr.hadoop</groupId>
+                        <artifactId>maprfs</artifactId>
+                        <version>${maprfs.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.hadoop</groupId>
+                        <artifactId>hadoop-client</artifactId>
+                        <version>${mapr-hadoop.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.hadoop</groupId>
+                        <artifactId>hadoop-auth</artifactId>
+                        <version>${mapr-hadoop.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.hbase</groupId>
+                        <artifactId>hbase-server</artifactId>
+                        <version>${mapr-hbase.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.hbase</groupId>
+                        <artifactId>hbase-testing-util</artifactId>
+                        <version>${mapr-hbase.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.hive</groupId>
+                        <artifactId>hive-exec</artifactId>
+                        <version>${mapr-hive.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.hive</groupId>
+                        <artifactId>hive-jdbc</artifactId>
+                        <version>${mapr-hive.version}</version>
+                    </dependency>
+
+                    <dependency>
+                        <groupId>org.apache.kafka</groupId>
+                        <artifactId>kafka_${scala.version}</artifactId>
+                        <version>${mapr-kafka.version}</version>
+                    </dependency>
+                    <dependency>
+                        <groupId>org.apache.storm</groupId>
+                        <artifactId>storm-kafka</artifactId>
+                        <version>${mapr-storm-kafka.version}</version>
+                        <exclusions>
+                            <exclusion>
+                                <groupId>ch.qos.logback</groupId>
+                                <artifactId>logback-classic</artifactId>
+                            </exclusion>
+                            <exclusion>
+                                <groupId>log4j</groupId>
+                                <artifactId>log4j</artifactId>
+                            </exclusion>
+                            <exclusion>
+                                <groupId>org.slf4j</groupId>
+                                <artifactId>log4j-over-slf4j</artifactId>
+                            </exclusion>
+                        </exclusions>
+                    </dependency>
                 </dependencies>
             </dependencyManagement>
         </profile>
@@ -950,33 +1070,14 @@
                                 <exclude>**/webapp/**</exclude>
 
                             </excludes>
-                         </configuration>
-                     </execution>
-                 </executions>
-             </plugin>
-         </plugins>
+                        </configuration>
+                    </execution>
+                </executions>
+            </plugin>
+        </plugins>
     </build>
     <repositories>
         <repository>
-            <id>HDP Release Repository</id>
-            <url>http://repo.hortonworks.com/content/repositories/releases/</url>
-            <releases>
-                <enabled>true</enabled>
-                <updatePolicy>never</updatePolicy>
-            </releases>
-        </repository>
-        <repository>
-            <id>HDP Central Repository</id>
-            <url>http://repo.hortonworks.com/content/repositories/central/</url>
-            <releases>
-                <enabled>true</enabled>
-                <updatePolicy>never</updatePolicy>
-                <checksumPolicy>warn</checksumPolicy>
-            </releases>
-            <name>HDP Central</name>
-            <layout>default</layout>
-        </repository>
-        <repository>
             <id>Maven Repo1 Repository</id>
             <url>http://repo1.maven.org/maven2</url>
             <releases>


Mime
View raw message