metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickal...@apache.org
Subject [5/5] metron git commit: METRON-1053 Relocate Metron Docker (nickwallen) closes apache/metron#659
Date Tue, 25 Jul 2017 17:11:47 GMT
METRON-1053 Relocate Metron Docker (nickwallen) closes apache/metron#659


Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/ba9ddda6
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/ba9ddda6
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/ba9ddda6

Branch: refs/heads/master
Commit: ba9ddda6d742a0df96e24a1573f44482020c25fb
Parents: 28a9740
Author: nickwallen <nick@nickallen.org>
Authored: Tue Jul 25 13:11:23 2017 -0400
Committer: nickallen <nickallen@apache.org>
Committed: Tue Jul 25 13:11:23 2017 -0400

----------------------------------------------------------------------
 README.md                                       |   4 +
 metron-contrib/metron-docker/.gitignore         |  13 +
 metron-contrib/metron-docker/README.md          | 255 +++++++++++++++
 .../metron-docker/compose/docker-compose.yml    |  82 +++++
 .../compose/elasticsearch/Dockerfile            |  27 ++
 .../compose/elasticsearch/docker-entrypoint.sh  |  37 +++
 .../metron-docker/compose/hadoop/Dockerfile     |  35 +++
 .../compose/hadoop/conf/core-site.xml           |  24 ++
 .../compose/hadoop/conf/hdfs-site.xml           |  32 ++
 .../compose/hadoop/docker-entrypoint.sh         |  38 +++
 .../metron-docker/compose/hbase/Dockerfile      |  43 +++
 .../compose/hbase/bin/init-commands.txt         |  22 ++
 .../compose/hbase/bin/init-hbase.sh             |  20 ++
 .../metron-docker/compose/hbase/bin/start.sh    |  21 ++
 .../hbase/conf/enrichment-extractor.json        |  12 +
 .../compose/hbase/conf/hbase-site.docker.xml    |  41 +++
 .../hbase/conf/threatintel-extractor.json       |  11 +
 .../compose/hbase/data/enrichments.csv          |   3 +
 .../compose/hbase/data/threatintel.csv          |   3 +
 .../metron-docker/compose/kafkazk/Dockerfile    |  44 +++
 .../compose/kafkazk/bin/create-topic.sh         |  18 ++
 .../compose/kafkazk/bin/init-kafka.sh           |  19 ++
 .../compose/kafkazk/bin/init-zk.sh              |  23 ++
 .../compose/kafkazk/bin/output-data.sh          |  37 +++
 .../compose/kafkazk/bin/produce-data.sh         |  28 ++
 .../compose/kafkazk/bin/run-consumer.sh         |  18 ++
 .../metron-docker/compose/kafkazk/bin/start.sh  |  24 ++
 .../compose/kafkazk/conf/global.json            |   6 +
 .../compose/kafkazk/data/BroExampleOutput.txt   |  10 +
 .../compose/kafkazk/data/SquidExampleOutput.txt |   5 +
 .../metron-docker/compose/kibana/Dockerfile     |  19 ++
 .../compose/kibana/conf/kibana-index.json       |  52 ++++
 .../compose/kibana/images/metron.svg            |  88 ++++++
 .../metron-docker/compose/storm/Dockerfile      |  56 ++++
 .../storm/bin/start_docker_parser_topology.sh   |  18 ++
 metron-contrib/metron-docker/conf/.env          |   3 +
 metron-contrib/metron-docker/install-metron.sh  |  36 +++
 metron-contrib/metron-docker/pom.xml            | 312 +++++++++++++++++++
 .../scripts/create-docker-machine.sh            |  18 ++
 .../metron-docker/scripts/wait-for-it.sh        | 161 ++++++++++
 metron-contrib/pom.xml                          |  44 +++
 metron-docker/.gitignore                        |  13 -
 metron-docker/README.md                         | 252 ---------------
 metron-docker/compose/docker-compose.yml        |  81 -----
 metron-docker/compose/elasticsearch/Dockerfile  |  27 --
 .../compose/elasticsearch/docker-entrypoint.sh  |  37 ---
 metron-docker/compose/hadoop/Dockerfile         |  34 --
 metron-docker/compose/hadoop/conf/core-site.xml |  24 --
 metron-docker/compose/hadoop/conf/hdfs-site.xml |  32 --
 .../compose/hadoop/docker-entrypoint.sh         |  38 ---
 metron-docker/compose/hbase/Dockerfile          |  43 ---
 .../compose/hbase/bin/init-commands.txt         |  22 --
 metron-docker/compose/hbase/bin/init-hbase.sh   |  20 --
 metron-docker/compose/hbase/bin/start.sh        |  21 --
 .../hbase/conf/enrichment-extractor.json        |  12 -
 .../compose/hbase/conf/hbase-site.docker.xml    |  41 ---
 .../hbase/conf/threatintel-extractor.json       |  11 -
 .../compose/hbase/data/enrichments.csv          |   3 -
 .../compose/hbase/data/threatintel.csv          |   3 -
 metron-docker/compose/kafkazk/Dockerfile        |  45 ---
 .../compose/kafkazk/bin/create-topic.sh         |  18 --
 metron-docker/compose/kafkazk/bin/init-kafka.sh |  19 --
 metron-docker/compose/kafkazk/bin/init-zk.sh    |  23 --
 .../compose/kafkazk/bin/output-data.sh          |  37 ---
 .../compose/kafkazk/bin/produce-data.sh         |  28 --
 .../compose/kafkazk/bin/run-consumer.sh         |  18 --
 metron-docker/compose/kafkazk/bin/start.sh      |  24 --
 metron-docker/compose/kafkazk/conf/global.json  |   6 -
 .../compose/kafkazk/data/BroExampleOutput.txt   |  10 -
 .../compose/kafkazk/data/SquidExampleOutput.txt |   5 -
 metron-docker/compose/kibana/Dockerfile         |  19 --
 .../compose/kibana/conf/kibana-index.json       |  52 ----
 metron-docker/compose/kibana/images/metron.svg  |  88 ------
 metron-docker/compose/storm/Dockerfile          |  56 ----
 .../storm/bin/start_docker_parser_topology.sh   |  18 --
 metron-docker/conf/.env                         |   3 -
 metron-docker/install-metron.sh                 |  36 ---
 metron-docker/pom.xml                           | 312 -------------------
 metron-docker/scripts/create-docker-machine.sh  |  18 --
 metron-docker/scripts/wait-for-it.sh            | 161 ----------
 pom.xml                                         |   2 +-
 81 files changed, 1763 insertions(+), 1711 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index dcea8fa..8599f0f 100644
--- a/README.md
+++ b/README.md
@@ -64,6 +64,10 @@ Metron's code:
 
 Option 3 is more likely to have the latest code.
 
+# Getting Started
+
+To start exploring the capabilities of Apache Metron [follow these instructions to launch Metron in a single-node VM](metron-deployment/vagrant/full-dev-platform) on your own hardware.  
+
 # Building Metron
 
 Build the full project and run tests:

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/.gitignore
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/.gitignore b/metron-contrib/metron-docker/.gitignore
new file mode 100644
index 0000000..05cbde5
--- /dev/null
+++ b/metron-contrib/metron-docker/.gitignore
@@ -0,0 +1,13 @@
+/compose/.env
+/compose/elasticsearch/es_templates
+/compose/kafkazk/packages/
+/compose/hbase/data-management
+/compose/storm/elasticsearch
+/compose/storm/enrichment
+/compose/storm/parser
+/compose/storm/indexing
+/compose/kafkazk/data/*
+!/compose/kafkazk/data/BroExampleOutput.txt
+!/compose/kafkazk/data/SquidExampleOutput.txt
+wait-for-it.sh
+!/scripts/wait-for-it.sh
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/README.md
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/README.md b/metron-contrib/metron-docker/README.md
new file mode 100644
index 0000000..54cb63b
--- /dev/null
+++ b/metron-contrib/metron-docker/README.md
@@ -0,0 +1,255 @@
+# Metron Docker
+
+Metron Docker is a [Docker Compose](https://docs.docker.com/compose/overview/) application that is intended only for development and integration testing of Metron.  These images can quickly spin-up the underlying components on which Apache Metron runs.
+
+None of the core Metron components are setup or launched automatically with these Docker images.  You will need to manually setup and start the Metron components that you require.  You should not expect to see telemetry being parsed, enriched, or indexed.  If you are looking to try-out, experiment or demo Metron capabilities on a single node, then the [Vagrant-driven VM](../../metron-deployment/vagrant/full-dev-platform) is what you need.  Use this instead of Vagrant when:
+  
+  - You want an environment that can be built and spun up quickly
+  - You need to frequently rebuild and restart services
+  - You only need to test, troubleshoot or develop against a subset of services
+  
+Metron Docker includes these images that have been customized for Metron:
+
+  - Kafka (with Zookeeper)
+  - HBase
+  - Storm
+  - Elasticsearch
+  - Kibana
+  - HDFS
+
+Setup
+-----
+
+Install [Docker for Mac](https://docs.docker.com/docker-for-mac/) or [Docker for Windows](https://docs.docker.com/docker-for-windows/).  The following versions have been tested:
+
+  - Docker version 1.12.0
+  - docker-machine version 0.8.0
+  - docker-compose version 1.8.0
+  
+Build Metron from the top level directory with:
+```
+$ cd $METRON_HOME
+$ mvn clean install -DskipTests
+```
+
+You are welcome to use an existing Docker host but we prefer one with more resources.  You can create one of those with this script:
+```
+$ export METRON_DOCKER_HOME=$METRON_HOME/metron-contrib/metron-docker
+$ cd $METRON_DOCKER_HOME 
+$ ./scripts/create-docker-machine.sh
+```
+
+This will create a host called "metron-machine".  Anytime you want to run Docker commands against this host, make sure you run this first to set the Docker environment variables:
+```
+$ eval "$(docker-machine env metron-machine)"
+```
+
+If you wish to use a local docker-engine install, please set an environment variable BROKER_IP_ADDR to the IP address of your host machine. This cannot be the loopback address.
+
+Usage
+-----
+
+Navigate to the compose application root:
+```
+$ cd $METRON_DOCKER_HOME/compose/
+```
+
+The Metron Docker environment lifecycle is controlled by the [docker-compose](https://docs.docker.com/compose/reference/overview/) command.  The service names can be found in the docker-compose.yml file.  For example, to build and start the environment run this command:
+```
+$ eval "$(docker-machine env metron-machine)"
+$ docker-compose up -d
+```
+
+After all services have started list the containers and ensure their status is 'Up':
+```
+$ docker ps --format 'table {{.Names}}\t{{.Status}}'
+NAMES                    STATUS
+metron_storm_1           Up 5 minutes
+metron_hbase_1           Up 5 minutes
+metron_kibana_1          Up 5 minutes
+metron_kafkazk_1         Up 5 minutes
+metron_elasticsearch_1   Up 5 minutes
+```
+
+Various services are exposed through http on the Docker host.  Get the host ip from the URL property:
+```
+$ docker-machine ls
+NAME             ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
+metron-machine   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.12.5
+```
+
+Then, assuming a host ip of `192.168.99.100`, the UIs and APIs are available at:
+
+* Storm - http://192.168.99.100:8080/
+* HBase - http://192.168.99.100:16010/
+* Elasticsearch - http://192.168.99.100:9200/_plugin/head/
+* Kibana - http://192.168.99.100:5601/
+* HDFS (Namenode) - http://192.168.99.100:50070/
+
+The Storm logs can be useful when troubleshooting topologies.  They can be found on the Storm container in `/usr/share/apache-storm/logs`.
+
+When done using the machine, shut it down with:
+```
+$ docker-compose down
+```
+
+Examples
+-----
+* [Deploy a new parser class](#deploy-a-new-parser-class)
+* [Connect to a container](#connect-to-a-container)
+* [Create a sensor from sample data](create-a-sensor-from-sample-data)
+* [Upload configs to Zookeeper](upload-configs-to-zookeeper)
+* [Manage a topology](manage-a-topology)
+* [Run sensor data end to end](run-sensor-data-end-to-end)
+
+
+### Deploy a new parser class
+
+After adding a new parser to metron-parsers, build Metron from the top level directory:
+```
+$ cd $METRON_HOME
+$ mvn clean install -DskipTests
+```
+
+Then run these commands to redeploy the parsers to the Storm image:
+```
+$ cd $METRON_DOCKER_HOME/compose
+$ docker-compose down
+$ docker-compose build storm
+$ docker-compose up -d
+```
+
+### Connect to a container
+
+Suppose there is a problem with Kafka and the logs are needed for further investigation. Run this command to connect and explore the running Kafka container:
+```
+$ cd $METRON_DOCKER_HOME/compose
+$ docker-compose exec kafkazk bash
+```
+
+### Create a sensor from sample data
+
+A tool for producing test data in Kafka is included with the Kafka/Zookeeper image.  It loops through lines in a test data file and outputs them to Kafka at the desired frequency.  Create a test data file in `./kafkazk/data/` and rebuild the Kafka/Zookeeper image:
+```
+$ cd $METRON_DOCKER_HOME/compose
+$ printf 'first test data\nsecond test data\nthird test data\n' > ./kafkazk/data/TestData.txt
+$ docker-compose down
+$ docker-compose build kafkazk
+$ docker-compose up -d
+```
+
+This will deploy the test data file to the Kafka/Zookeeper container.  Now that data can be streamed to a Kafka topic:
+```
+$ docker-compose exec kafkazk ./bin/produce-data.sh
+Usage:  produce-data.sh data_path topic [message_delay_in_seconds]
+
+# Stream data in TestData.txt to the 'test' Kafka topic at a frequency of 5 seconds (default is 1 second)
+$ docker-compose exec kafkazk ./bin/produce-data.sh /data/TestData.txt test 5 
+```
+
+The Kafka/Zookeeper image comes with sample Bro and Squid data:
+```
+# Stream Bro test data every 1 second
+$ docker-compose exec kafkazk ./bin/produce-data.sh /data/BroExampleOutput.txt bro
+
+# Stream Squid test data every 0.1 seconds
+$ docker-compose exec kafkazk ./bin/produce-data.sh /data/SquidExampleOutput.txt squid 0.1
+```
+
+### Upload configs to Zookeeper
+
+Parser configs and a global config configured for this Docker environment are included with the Kafka/Zookeeper image.  Load them with:
+```
+$ docker-compose exec kafkazk bash
+# $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i $METRON_HOME/config/zookeeper
+# exit
+```
+
+Dump out the configs with:
+```
+$ docker-compose exec kafkazk bash
+# $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m DUMP
+# exit
+```
+
+### Manage a topology
+
+The Storm image comes with a script to easily start parser topologies:
+```
+docker-compose exec storm ./bin/start_docker_parser_topology.sh sensor_name
+```
+
+The enrichment topology can be started with:
+```
+docker-compose exec storm ./bin/start_enrichment_topology.sh
+```
+
+The indexing topology can be started with:
+```
+docker-compose exec storm ./bin/start_elasticsearch_topology.sh
+```
+
+Topologies can be stopped using the Storm CLI.  For example, stop the enrichment topology with:
+```
+docker-compose exec storm storm kill enrichments -w 0
+```
+
+### Run sensor data end to end
+
+First ensure configs were uploaded as described in the previous example. Then start a sensor and leave it running:
+```
+$ cd $METRON_DOCKER_HOME/compose
+$ docker-compose exec kafkazk ./bin/produce-data.sh /data/BroExampleOutput.txt bro
+```
+
+Open a separate console session and verify the sensor is running by consuming a message from Kafka:
+```
+$ export METRON_DOCKER_HOME=$METRON_HOME/metron-contrib/metron-docker
+$ cd $METRON_DOCKER_HOME/compose
+$ docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
+```
+
+A new message should be printed every second. Now kill the consumer and start the Bro parser topology:
+```
+$ docker-compose exec storm ./bin/start_docker_parser_topology.sh bro
+```
+
+Bro data should be flowing through the bro parser topology and into the Kafka enrichments topic.  The enrichments topic should be created automatically:
+```
+$ docker-compose exec kafkazk ./bin/kafka-topics.sh --zookeeper localhost:2181 --list
+bro
+enrichments
+indexing
+```
+
+Verify parsed Bro data is in the Kafka enrichments topic:
+```
+docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic enrichments
+```
+
+Now start the enrichment topology:
+```
+docker-compose exec storm ./bin/start_enrichment_topology.sh
+```
+
+Parsed Bro data should be flowing through the enrichment topology and into the Kafka indexing topic.  Verify enriched Bro data is in the Kafka indexing topic:
+```
+docker-compose exec kafkazk ./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic indexing
+```
+
+Now start the indexing topology:
+```
+docker-compose exec storm ./bin/start_elasticsearch_topology.sh
+```
+
+Enriched Bro data should now be present in the Elasticsearch container:
+```
+$ docker-machine ls
+NAME             ACTIVE   DRIVER       STATE     URL                         SWARM   DOCKER    ERRORS
+metron-machine   *        virtualbox   Running   tcp://192.168.99.100:2376           v1.12.5
+
+$ curl -XGET http://192.168.99.100:9200/_cat/indices?v
+health status index                   pri rep docs.count docs.deleted store.size pri.store.size
+yellow open   .kibana                   1   1          1            0      3.1kb          3.1kb
+yellow open   bro_index_2016.12.19.18   5   1        180            0      475kb          475kb
+```
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/docker-compose.yml
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/docker-compose.yml b/metron-contrib/metron-docker/compose/docker-compose.yml
new file mode 100644
index 0000000..5831b44
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/docker-compose.yml
@@ -0,0 +1,82 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+version: '2'
+services:
+  kafkazk:
+    build:
+      context: ./kafkazk
+      args:
+        DOCKER_HOST: $DOCKER_HOST
+        BROKER_IP_ADDR: $BROKER_IP_ADDR
+        METRON_VERSION: $METRON_VERSION
+    ports:
+      - "9092:9092"
+      - "2181:2181"
+  hbase:
+    build:
+      context: ./hbase
+      args:
+        METRON_VERSION: $METRON_VERSION
+    ports:
+      - "16010:16010"
+    volumes:
+      - "/opt/hbase/conf"
+    depends_on:
+      - kafkazk
+  hadoop:
+    build:
+      context: ./hadoop
+    ports:
+      - "50070:50070"
+    volumes:
+      - "/opt/hadoop/etc/hadoop"
+    command: tail -f /dev/null
+  storm:
+    build:
+      context: ./storm
+      args:
+        METRON_VERSION: $METRON_VERSION
+    ports:
+      - "8000:8000"
+      - "8080:8080"
+      - "8081:8081"
+    environment:
+      ZOOKEEPER_ADDR: kafkazk
+      CONFIG_TOPOLOGY_CLASSPATH: "/opt/hbase/conf:/opt/hadoop/etc/hadoop"
+      CONFIG_TOPOLOGY_CLASSPATH: "/opt/hbase/conf:/opt/hadoop/etc/hadoop"
+    volumes_from:
+      - hbase
+      - hadoop
+    depends_on:
+      - kafkazk
+      - hbase
+      - elasticsearch
+      - hadoop
+    command: --daemon nimbus supervisor ui logviewer
+  elasticsearch:
+    build:
+      context: ./elasticsearch
+    ports:
+      - "9200:9200"
+      - "9300:9300"
+    command: tail -f /dev/null
+  kibana:
+    build: ./kibana
+    ports:
+      - "5601:5601"
+    depends_on:
+      - elasticsearch

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/elasticsearch/Dockerfile
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/elasticsearch/Dockerfile b/metron-contrib/metron-docker/compose/elasticsearch/Dockerfile
new file mode 100644
index 0000000..790d9eb
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/elasticsearch/Dockerfile
@@ -0,0 +1,27 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+FROM elasticsearch:2.3
+
+RUN /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
+
+ADD ./es_templates /es_templates
+
+COPY ./wait-for-it.sh /
+RUN chmod 755 /wait-for-it.sh
+
+COPY ./docker-entrypoint.sh /
+ENTRYPOINT ["/docker-entrypoint.sh"]

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/elasticsearch/docker-entrypoint.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/elasticsearch/docker-entrypoint.sh b/metron-contrib/metron-docker/compose/elasticsearch/docker-entrypoint.sh
new file mode 100755
index 0000000..00209dc
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/elasticsearch/docker-entrypoint.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# exit immediately on error
+set -e
+
+# start elasticsearch as non-root user
+chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data
+chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/logs
+gosu elasticsearch /usr/share/elasticsearch/bin/elasticsearch -d
+
+# wait for elasticsearch to start
+/wait-for-it.sh localhost:9200 -t 30
+
+# load elasticsearch templates
+for template_file in `ls -1 /es_templates`; do
+    template_name=`echo $template_file | sed 's/\.template//g'`
+    curl -XPUT --data @/es_templates/$template_file http://localhost:9200/_template/$template_name
+done
+
+# pass through CMD as PID 1
+exec "$@"

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hadoop/Dockerfile
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hadoop/Dockerfile b/metron-contrib/metron-docker/compose/hadoop/Dockerfile
new file mode 100644
index 0000000..9a0a3ce
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hadoop/Dockerfile
@@ -0,0 +1,35 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+FROM openjdk:7
+
+ENV HADOOP_PREFIX=/opt/hadoop
+ENV HADOOP_CONF_DIR=$HADOOP_PREFIX/etc/hadoop
+ENV HADOOP_IDENT_STRING=root
+
+RUN curl -sL http://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz | tar -xzC /tmp
+RUN mv /tmp/hadoop-2.7.3 /opt/hadoop
+
+COPY ./conf/* $HADOOP_CONF_DIR/
+RUN mkdir -p /hadoop/dfs/name \
+    && mkdir -p /hadoop/dfs/data \
+    && $HADOOP_PREFIX/bin/hdfs namenode -format
+
+WORKDIR $HADOOP_PREFIX
+EXPOSE 9000 50010 50020 50070 50075
+
+COPY ./docker-entrypoint.sh /
+ENTRYPOINT ["/docker-entrypoint.sh"]

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hadoop/conf/core-site.xml
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hadoop/conf/core-site.xml b/metron-contrib/metron-docker/compose/hadoop/conf/core-site.xml
new file mode 100644
index 0000000..f7a9465
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hadoop/conf/core-site.xml
@@ -0,0 +1,24 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+    <property>
+        <name>fs.defaultFS</name>
+        <value>hdfs://hadoop:9000</value>
+    </property>
+</configuration>

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hadoop/conf/hdfs-site.xml
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hadoop/conf/hdfs-site.xml b/metron-contrib/metron-docker/compose/hadoop/conf/hdfs-site.xml
new file mode 100644
index 0000000..c75a464
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hadoop/conf/hdfs-site.xml
@@ -0,0 +1,32 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+<!-- Put site-specific property overrides in this file. -->
+
+<configuration>
+    <property>
+        <name>dfs.replication</name>
+        <value>1</value>
+    </property>
+    <property>
+        <name>dfs.namenode.name.dir</name>
+        <value>/hadoop/dfs/name</value>
+    </property>
+    <property>
+        <name>dfs.datanode.data.dir</name>
+        <value>/hadoop/dfs/data</value>
+    </property>
+</configuration>

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hadoop/docker-entrypoint.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hadoop/docker-entrypoint.sh b/metron-contrib/metron-docker/compose/hadoop/docker-entrypoint.sh
new file mode 100755
index 0000000..97b9809
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hadoop/docker-entrypoint.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# exit immediately on error
+set -e
+
+# start namenode
+$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_PREFIX/etc/hadoop --script hdfs start namenode
+
+# start datanode
+$HADOOP_PREFIX/sbin/hadoop-daemon.sh --config $HADOOP_PREFIX/etc/hadoop --script hdfs start datanode
+
+# create metron base directory
+$HADOOP_PREFIX/bin/hdfs dfs -mkdir -p /apps/metron
+
+# create directory for geo database
+$HADOOP_PREFIX/bin/hdfs dfs -mkdir -p /apps/metron/geo/default
+
+# download geo database to hdfs
+curl http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz | $HADOOP_PREFIX/bin/hdfs dfs -put - /apps/metron/geo/default/GeoLite2-City.mmdb.gz
+
+# pass through CMD as PID 1
+exec "$@"

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/Dockerfile
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/Dockerfile b/metron-contrib/metron-docker/compose/hbase/Dockerfile
new file mode 100644
index 0000000..bab2e67
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/Dockerfile
@@ -0,0 +1,43 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+FROM centos
+
+ARG METRON_VERSION
+
+ENV METRON_VERSION $METRON_VERSION
+ENV JAVA_HOME /usr
+ENV HBASE_HOME /opt/hbase
+ENV HBASE_MANAGES_ZK false
+ENV METRON_HOME /usr/metron/$METRON_VERSION/
+
+ADD ./data /data
+ADD ./data-management /data-management
+RUN mkdir -p $METRON_HOME
+RUN tar -xzf /data-management/metron-data-management-$METRON_VERSION-archive.tar.gz -C /usr/metron/$METRON_VERSION/
+RUN curl -sL http://archive.apache.org/dist/hbase/1.1.6/hbase-1.1.6-bin.tar.gz | tar -xzC /tmp
+RUN mv /tmp/hbase-1.1.6 /opt/hbase
+RUN yum install -y java-1.8.0-openjdk lsof
+ADD ./conf/enrichment-extractor.json /conf/enrichment-extractor.json
+ADD ./conf/threatintel-extractor.json /conf/threatintel-extractor.json
+ADD ./conf/hbase-site.docker.xml $HBASE_HOME/conf/hbase-site.xml
+ADD ./bin $HBASE_HOME/bin
+RUN chmod 755 $HBASE_HOME/bin/wait-for-it.sh
+
+EXPOSE 8080 8085 9090 9095 16000 16010 16201 16301
+
+WORKDIR /opt/hbase
+CMD ./bin/start.sh

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/bin/init-commands.txt
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/bin/init-commands.txt b/metron-contrib/metron-docker/compose/hbase/bin/init-commands.txt
new file mode 100755
index 0000000..9bdf61f
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/bin/init-commands.txt
@@ -0,0 +1,22 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+create 'access_tracker', 'cf'
+create 'ip', 'cf'
+create 'enrichment', 'cf'
+create 'threatintel', 'cf'
+exit
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/bin/init-hbase.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/bin/init-hbase.sh b/metron-contrib/metron-docker/compose/hbase/bin/init-hbase.sh
new file mode 100755
index 0000000..8b2d1d3
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/bin/init-hbase.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+./bin/hbase shell ./bin/init-commands.txt
+/usr/metron/$METRON_VERSION/bin/flatfile_loader.sh -e /conf/enrichment-extractor.json -t enrichment -c cf -i /data/enrichments.csv
+/usr/metron/$METRON_VERSION/bin/flatfile_loader.sh -e /conf/threatintel-extractor.json -t threatintel -c cf -i /data/threatintel.csv

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/bin/start.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/bin/start.sh b/metron-contrib/metron-docker/compose/hbase/bin/start.sh
new file mode 100755
index 0000000..20d01d2
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/bin/start.sh
@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+./bin/start-hbase.sh
+./bin/wait-for-it.sh localhost:16010
+./bin/init-hbase.sh
+tail -f /dev/null

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/conf/enrichment-extractor.json
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/conf/enrichment-extractor.json b/metron-contrib/metron-docker/compose/hbase/conf/enrichment-extractor.json
new file mode 100644
index 0000000..322dbfd
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/conf/enrichment-extractor.json
@@ -0,0 +1,12 @@
+{
+  "config" : {
+    "columns" : {
+      "ip" : 0
+    ,"message" : 1
+    }
+  ,"indicator_column" : "ip"
+  ,"type" : "sample"
+  ,"separator" : ","
+  }
+,"extractor" : "CSV"
+}

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/conf/hbase-site.docker.xml
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/conf/hbase-site.docker.xml b/metron-contrib/metron-docker/compose/hbase/conf/hbase-site.docker.xml
new file mode 100644
index 0000000..16a9c65
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/conf/hbase-site.docker.xml
@@ -0,0 +1,41 @@
+<?xml version="1.0"?>
+<!--
+  Licensed to the Apache Software
+	Foundation (ASF) under one or more contributor license agreements. See the
+	NOTICE file distributed with this work for additional information regarding
+	copyright ownership. The ASF licenses this file to You under the Apache License,
+	Version 2.0 (the "License"); you may not use this file except in compliance
+	with the License. You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software distributed
+	under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+	OR CONDITIONS OF ANY KIND, either express or implied. See the License for
+  the specific language governing permissions and limitations under the License.
+  -->
+<configuration>
+    <property>
+        <name>hbase.rootdir</name>
+        <value>file:///home/root/hbase</value>
+    </property>
+    <property>
+        <name>hbase.zookeeper.property.dataDir</name>
+        <value>/home/root/zookeeper</value>
+    </property>
+
+    <property>
+        <name>hbase.zookeeper.property.clientPort</name>
+        <value>2181</value>
+        <description>Property from ZooKeeper's config zoo.cfg.
+            The port at which the clients will connect.
+        </description>
+    </property>
+
+    <property>
+        <name>hbase.zookeeper.quorum</name>
+        <value>kafkazk</value>
+        <description>Comma separated list of servers in the ZooKeeper Quorum.</description>
+    </property>
+
+</configuration>

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/conf/threatintel-extractor.json
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/conf/threatintel-extractor.json b/metron-contrib/metron-docker/compose/hbase/conf/threatintel-extractor.json
new file mode 100644
index 0000000..9e32d67
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/conf/threatintel-extractor.json
@@ -0,0 +1,11 @@
+{
+  "config": {
+    "columns": {
+      "ip": 0
+    },
+    "indicator_column": "ip",
+    "type" : "malicious_ip",
+    "separator": ","
+  },
+  "extractor": "CSV"
+}

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/data/enrichments.csv
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/data/enrichments.csv b/metron-contrib/metron-docker/compose/hbase/data/enrichments.csv
new file mode 100644
index 0000000..4db096d
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/data/enrichments.csv
@@ -0,0 +1,3 @@
+93.188.160.43,enrichment 1
+192.249.113.37,enrichment 2
+10.122.196.204,enrichment 3
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/hbase/data/threatintel.csv
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/hbase/data/threatintel.csv b/metron-contrib/metron-docker/compose/hbase/data/threatintel.csv
new file mode 100644
index 0000000..e68913c
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/hbase/data/threatintel.csv
@@ -0,0 +1,3 @@
+93.188.160.43
+192.249.113.37
+10.122.196.204
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/Dockerfile
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/Dockerfile b/metron-contrib/metron-docker/compose/kafkazk/Dockerfile
new file mode 100644
index 0000000..ebade48
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/Dockerfile
@@ -0,0 +1,44 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+FROM centos
+
+ARG DOCKER_HOST
+ARG BROKER_IP_ADDR
+ARG METRON_VERSION
+
+ENV METRON_VERSION $METRON_VERSION
+ENV METRON_HOME /usr/metron/$METRON_VERSION/
+ENV ZK_CLIENT_JARS /opt/kafka/libs
+
+RUN curl -sL https://archive.apache.org/dist/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz | tar -xzC /tmp
+RUN mv /tmp/kafka_2.11-0.10.0.0 /opt/kafka
+RUN echo -n 'advertised.listeners=PLAINTEXT://' >> /opt/kafka/config/server.properties
+RUN echo $DOCKER_HOST | sed "s/^$/"$BROKER_IP_ADDR":/g" | sed "s/tcp:\\/\\///g" | sed "s/:.*/:9092/g" >> /opt/kafka/config/server.properties
+RUN echo 'delete.topic.enable=true' >> /opt/kafka/config/server.properties
+RUN yum install -y java-1.8.0-openjdk lsof
+RUN mkdir -p $METRON_HOME
+ADD ./bin /opt/kafka/bin
+RUN chmod 755 /opt/kafka/bin/wait-for-it.sh
+COPY ./data /data/
+COPY ./packages/* /packages/
+RUN find /packages -type f -name '*.tar.gz' -exec tar -xzf {} -C /usr/metron/$METRON_VERSION/ \;
+ADD ./conf /$METRON_HOME/config/zookeeper
+
+EXPOSE 2181 9092
+
+WORKDIR /opt/kafka
+CMD ./bin/start.sh

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/bin/create-topic.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/bin/create-topic.sh b/metron-contrib/metron-docker/compose/kafkazk/bin/create-topic.sh
new file mode 100755
index 0000000..7db950a
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/bin/create-topic.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic $1

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/bin/init-kafka.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/bin/init-kafka.sh b/metron-contrib/metron-docker/compose/kafkazk/bin/init-kafka.sh
new file mode 100755
index 0000000..078c184
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/bin/init-kafka.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+./bin/create-topic.sh enrichments
+./bin/create-topic.sh indexing

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/bin/init-zk.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/bin/init-zk.sh b/metron-contrib/metron-docker/compose/kafkazk/bin/init-zk.sh
new file mode 100755
index 0000000..4974683
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/bin/init-zk.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+echo "create /metron metron" | ./bin/zookeeper-shell.sh localhost:2181
+echo "create /metron/topology topology" | ./bin/zookeeper-shell.sh localhost:2181
+echo "create /metron/topology/parsers parsers" | ./bin/zookeeper-shell.sh localhost:2181
+echo "create /metron/topology/enrichments enrichments" | ./bin/zookeeper-shell.sh localhost:2181
+echo "create /metron/topology/indexing indexing" | ./bin/zookeeper-shell.sh localhost:2181
+$METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i $METRON_HOME/config/zookeeper

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/bin/output-data.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/bin/output-data.sh b/metron-contrib/metron-docker/compose/kafkazk/bin/output-data.sh
new file mode 100755
index 0000000..1ed3b37
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/bin/output-data.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+trap trapint 2
+function trapint {
+    exit 0
+}
+if [ $# -ne 2 ]
+  then
+    echo "Usage: output-data.sh data_path [message_delay_in_seconds]"
+    exit 0
+fi
+
+FILE_PATH=$1
+DELAY=$2
+while :
+do
+cat $FILE_PATH | while read line
+do
+echo "$line"
+sleep $DELAY
+done
+done

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/bin/produce-data.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/bin/produce-data.sh b/metron-contrib/metron-docker/compose/kafkazk/bin/produce-data.sh
new file mode 100755
index 0000000..e12b1bb
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/bin/produce-data.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+if [ $# -lt 2 ]
+  then
+    echo "Usage:  produce-data.sh data_path topic [message_delay_in_seconds]"
+    exit 0
+fi
+
+FILE_PATH=$1
+TOPIC=$2
+DELAY=${3:-1}
+echo "Emitting data in $FILE_PATH to Kafka topic $TOPIC every $DELAY second(s)"
+exec ./bin/output-data.sh $FILE_PATH $DELAY | ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic $TOPIC > /dev/null

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/bin/run-consumer.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/bin/run-consumer.sh b/metron-contrib/metron-docker/compose/kafkazk/bin/run-consumer.sh
new file mode 100755
index 0000000..af744d9
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/bin/run-consumer.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+./bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic $1

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/bin/start.sh
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/bin/start.sh b/metron-contrib/metron-docker/compose/kafkazk/bin/start.sh
new file mode 100755
index 0000000..757e0e6
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/bin/start.sh
@@ -0,0 +1,24 @@
+#!/bin/bash
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+./bin/zookeeper-server-start.sh config/zookeeper.properties &
+./bin/wait-for-it.sh localhost:2181
+./bin/init-zk.sh
+./bin/kafka-server-start.sh config/server.properties &
+./bin/wait-for-it.sh localhost:9092
+./bin/init-kafka.sh
+tail -f /dev/null

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/conf/global.json
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/conf/global.json b/metron-contrib/metron-docker/compose/kafkazk/conf/global.json
new file mode 100644
index 0000000..4a1e302
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/conf/global.json
@@ -0,0 +1,6 @@
+{
+  "es.clustername": "elasticsearch",
+  "es.ip": "elasticsearch",
+  "es.port": "9300",
+  "es.date.format": "yyyy.MM.dd.HH"
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/data/BroExampleOutput.txt
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/data/BroExampleOutput.txt b/metron-contrib/metron-docker/compose/kafkazk/data/BroExampleOutput.txt
new file mode 100644
index 0000000..d6ab902
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/data/BroExampleOutput.txt
@@ -0,0 +1,10 @@
+{"http":{"ts":1402307733,"uid":"CTo78A11g7CYbbOHvj","id.orig_h":"192.249.113.37","id.orig_p":58808,"id.resp_h":"72.163.4.161","id.resp_p":80,"trans_depth":1,"method":"GET","host":"www.cisco.com","uri":"/","user_agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3","request_body_len":0,"response_body_len":25523,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FJDyMC15lxUn5ngPfd"],"resp_mime_types":["text/html"]}}
+{"dns":{"ts":1402308259,"uid":"CuJT272SKaJSuqO0Ia","id.orig_h":"10.122.196.204","id.orig_p":33976,"id.resp_h":"144.254.71.184","id.resp_p":53,"proto":"udp","trans_id":62418,"query":"www.cisco.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["www.cisco.com.akadns.net","origin-www.cisco.com","2001:420:1201:2::a"],"TTLs":[3600.0,289.0,14.0],"rejected":false}}
+{"http":{"ts":1402307733,"uid":"KIRAN","id.orig_h":"10.122.196.204","id.orig_p":58808,"id.resp_h":"72.163.4.161","id.resp_p":80,"trans_depth":1,"method":"GET","host":"www.cisco.com","uri":"/","user_agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3","request_body_len":0,"response_body_len":25523,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FJDyMC15lxUn5ngPfd"],"resp_mime_types":["text/html"]}}
+{"http":{"ts":1402307733,"uid":"KIRAN12312312","id.orig_h":"192.249.113.37","id.orig_p":58808,"id.resp_h":"72.163.4.161","id.resp_p":80,"trans_depth":1,"method":"GET","host":"www.cisco.com","uri":"/","user_agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3","request_body_len":0,"response_body_len":25523,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FJDyMC15lxUn5ngPfd"],"resp_mime_types":["text/html"]}}
+{"http":{"ts":1402307733,"uid":"KIRAN12312312","id.orig_h":"192.249.113.37","id.orig_p":58808,"id.resp_h":"72.163.4.161","id.resp_p":80,"trans_depth":1,"method":"GET","host":"www.cisco.com","uri":"/","user_agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3","request_body_len":0,"response_body_len":25523,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FJDyMC15lxUn5ngPfd"],"resp_mime_types":["text/html"]}}
+{"http":{"ts":1402307733,"uid":"CTo78A11g7CYbbOHvj","id.orig_h":"10.122.196.204","id.orig_p":58808,"id.resp_h":"72.163.4.161","id.resp_p":80,"trans_depth":1,"email":"abullis@mail.csuchico.edu","method":"GET","host":"gabacentre.pw","uri":"/","user_agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3","request_body_len":0,"response_body_len":25523,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FJDyMC15lxUn5ngPfd"],"resp_mime_types":["text/html"]}}
+{"dns":{"ts":1402308259,"uid":"CYbbOHvj","id.orig_h":"93.188.160.43","id.orig_p":33976,"id.resp_h":"144.254.71.184","id.resp_p":53,"proto":"udp","trans_id":62418,"query":"www.cisco.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["gabacentre.pw","www.cisco.com.akadns.net","origin-www.cisco.com","2001:420:1201:2::a"],"TTLs":[3600.0,289.0,14.0],"rejected":false}}
+{"http":{"ts":1402307733,"uid":"CTo78A11g7CYbbOHvj","id.orig_h":"192.249.113.37","id.orig_p":58808,"id.resp_h":"72.163.4.161","id.resp_p":80,"trans_depth":1,"method":"GET","host":"www.cisco.com","uri":"/","user_agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3","request_body_len":0,"response_body_len":25523,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FJDyMC15lxUn5ngPfd"],"resp_mime_types":["text/html"]}}
+{"dns":{"ts":1402308259,"uid":"CuJT272SKaJSuqO0Ia","id.orig_h":"10.122.196.204","id.orig_p":33976,"id.resp_h":"144.254.71.184","id.resp_p":53,"proto":"udp","trans_id":62418,"query":"www.cisco.com","qclass":1,"qclass_name":"C_INTERNET","qtype":28,"qtype_name":"AAAA","rcode":0,"rcode_name":"NOERROR","AA":true,"TC":false,"RD":true,"RA":true,"Z":0,"answers":["www.cisco.com.akadns.net","origin-www.cisco.com","2001:420:1201:2::a"],"TTLs":[3600.0,289.0,14.0],"rejected":false}}
+{"http":{"ts":1402307733,"uid":"KIRAN","id.orig_h":"10.122.196.204","id.orig_p":58808,"id.resp_h":"72.163.4.161","id.resp_p":80,"trans_depth":1,"method":"GET","host":"www.cisco.com","uri":"/","user_agent":"curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3","request_body_len":0,"response_body_len":25523,"status_code":200,"status_msg":"OK","tags":[],"resp_fuids":["FJDyMC15lxUn5ngPfd"],"resp_mime_types":["text/html"]}}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kafkazk/data/SquidExampleOutput.txt
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kafkazk/data/SquidExampleOutput.txt b/metron-contrib/metron-docker/compose/kafkazk/data/SquidExampleOutput.txt
new file mode 100644
index 0000000..358a24d
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kafkazk/data/SquidExampleOutput.txt
@@ -0,0 +1,5 @@
+1461576382.642 161 127.0.0.1 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
+1461576442.228 159 127.0.0.1 TCP_MISS/200 137183 GET http://www.nba.com/ - DIRECT/66.210.41.9 text/html
+1467011157.401 415 127.0.0.1 TCP_MISS/200 337891 GET http://www.aliexpress.com/af/shoes.html? - DIRECT/207.109.73.154 text/html
+1467011158.083 671 127.0.0.1 TCP_MISS/200 41846 GET http://www.help.1and1.co.uk/domains-c40986/transfer-domains-c79878 - DIRECT/212.227.34.3 text/html
+1467011159.978 1893 127.0.0.1 TCP_MISS/200 153925 GET http://www.pravda.ru/science/ - DIRECT/185.103.135.90 text/html
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/ba9ddda6/metron-contrib/metron-docker/compose/kibana/Dockerfile
----------------------------------------------------------------------
diff --git a/metron-contrib/metron-docker/compose/kibana/Dockerfile b/metron-contrib/metron-docker/compose/kibana/Dockerfile
new file mode 100644
index 0000000..3bd0640
--- /dev/null
+++ b/metron-contrib/metron-docker/compose/kibana/Dockerfile
@@ -0,0 +1,19 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+FROM kibana:4.5.3
+
+ADD /images/metron.svg /opt/kibana/optimize/bundles/src/ui/public/images/kibana.svg


Mime
View raw message