Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 62C09200C3B for ; Fri, 10 Feb 2017 21:21:45 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 616C2160B4E; Fri, 10 Feb 2017 20:21:45 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 529D7160B71 for ; Fri, 10 Feb 2017 21:21:43 +0100 (CET) Received: (qmail 11842 invoked by uid 500); 10 Feb 2017 20:21:42 -0000 Mailing-List: contact commits-help@metron.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.incubator.apache.org Delivered-To: mailing list commits@metron.incubator.apache.org Received: (qmail 11812 invoked by uid 99); 10 Feb 2017 20:21:42 -0000 Received: from Unknown (HELO svn01-us-west.apache.org) (209.188.14.144) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Feb 2017 20:21:42 +0000 Received: from svn01-us-west.apache.org (localhost [127.0.0.1]) by svn01-us-west.apache.org (ASF Mail Server at svn01-us-west.apache.org) with ESMTP id 1A2A33A3221 for ; Fri, 10 Feb 2017 20:21:41 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r18274 [6/12] - in /dev/incubator/metron/0.3.1-RC4-incubating: book-site/ book-site/css/ book-site/image-archive/ book-site/images/ book-site/images/logos/ book-site/images/profiles/ book-site/img/ book-site/js/ book-site/metron-analytics/ ... Date: Fri, 10 Feb 2017 20:21:40 -0000 To: commits@metron.apache.org From: cestella@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20170210202141.1A2A33A3221@svn01-us-west.apache.org> archived-at: Fri, 10 Feb 2017 20:21:45 -0000 Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packaging/docker/ansible-docker/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packaging/docker/ansible-docker/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packaging/docker/ansible-docker/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,239 @@ + + + + + + + + + Metron – Overview + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Overview

+

+

The Metron ansible-docker container is provided in an effort reduce the installation burden of deploying Metron in a live envirionment. It is provisioned with software required to sucessfully run the deployment scripts.

+
+

Building the Container

+ +
    + +
  1. Install Docker [https://www.docker.com/products/overview]
    2. + +
  2. Navigate to <project-directory>/metron-deployment/packaging/docker/ansible-docker
    3. + +
  3. Build the container docker build -t ansible-docker:2.0.0.2 .
    4. +
+
+

Using the Container

+

Full instructions are found on the wiki [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361].

+

tl;dr

+ +
    + +
  1. docker run -it -v <project-directory>:/root/incubator-metron ansible-docker:2.0.0.2 bash
    2. + +
  2. cd /root/incubator-metron
    3. + +
  3. mvn clean package -DskipTests
    4. +
+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packaging/docker/rpm-docker/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packaging/docker/rpm-docker/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packaging/docker/rpm-docker/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,226 @@ + + + + + + + + + Metron – Overview + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Overview

+

+

The Metron ansible-docker container is provided in an effort reduce the installation burden of deploying Metron in a live envirionment. It is provisioned with software required to sucessfully run the deployment scripts.

+
+

Building the Container

+ +
    + +
  1. Install Docker [https://www.docker.com/products/overview]
    2. + +
  2. Navigate to <project-directory>/metron-deployment/packaging/rpm-docker
    3. + +
  3. Build the container docker build -t rpm-docker .
    4. +
+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packer-build/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packer-build/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/packer-build/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,274 @@ + + + + + + + + + Metron – Build Metron Images + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Build Metron Images

+

Based on the fantastic Bento project developed by Chef.

+
+

Images Provided

+ +
    + +
  • hdp-centos-6.7: Centos 6.7 + HDP. Used in the quick-dev-platform Vagrant image
    • + +
  • metron-centos-6.7: Centos 6.7 + HDP + Metron. Used for the codelab-platform Vagrant image.
    • +
+
+

Prerequisites

+ +
    + +
  • Packer 0.10.1
    • + +
  • Virtualbox 5.0.16
    • + +
  • Be sure to build Metron prior to building the images (cd your-project-directory/metron-platform && mvn clean package -DskipTests)
    • +
+

Build Both Images ———————- Navigate to <your-project-directory>/metron-deployment/packer-build Execute bin/bento build

+

Packer will build both images and export .box files to the ./builds directory.

+

Build Single Images ———————- Navigate to your-project-directory/metron-deployment/packer-build

+ +
    + +
  • HDP Centos
    • +
+ +
+
+
bin/bento build hdp-centos-6.7.json
+
+ +
    + +
  • Full Metron
    • +
+ +
+
+
bin/bento build metron-centos-6.7.json
+
+

Using Your New Box File ———————- Modify the relevant Vagrantfile (codelab-platform or quick-dev-platform) replacing the lines:

+ +
+
+
<pre><code>config.vm.box = "<i>box_name</i>"
+config.ssh.insert_key = true</code></pre>
+
+

with

+ +
+
+
<pre></code>config.vm.box = "<i>test_box_name</i>"
+config.vm.box = "<i>PathToBoxfile/Boxfilename</i>"
+config.ssh.insert_key = true</code></pre>
+
+

Launch the image as usual.

+

Node: Vagrant will cache boxes, you can force Vagrant to reload your box by running vagrant box remove test_box_name before launching your new image.

+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,271 @@ + + + + + + + + + Metron – Ansible Roles + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Ansible Roles

+

+ +
    + +
  • Kibana
    • + +
  • Monit
    • + +
  • OpenTaxii
    • + +
  • Pcap Relay
    • + +
  • Sensor Test Mode
    • +
+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/kibana/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/kibana/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/kibana/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,285 @@ + + + + + + + + + Metron – Kibana 4 + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +
+

Kibana 4

+

This role installs Kibana along with the default Metron Dashboard.

+
+

FAQ

+
+

How do I change Metron’s default dashboard?

+

Kibana stores all configuration elements within an Elasticsearch index called .kibana. To deploy Kibana in a desired state, including the Metron Dashboard, we simply take an extract from a functioning Kibana instance and store that in templates/kibana-index.json. The deployment process then restores the index from this extract.

+

(1) Stand-up an instance of Apache Metron and create the Kibana index patterns, visualizations, and dashboard as you see fit.

+

(2) Run the following command to extract the definitions for all the components that you have created. Be sure to delete anything that you don’t want to be part of this extract. It will include all artifacts present in your .kibana index.

+ +
+
+
  elasticdump --input=http://ec2-52-41-121-175.us-west-2.compute.amazonaws.com:9200/.kibana \
+    --output=kibana-index.json \
+    --type=data \
+    --searchBody='{"filter": { "or": [ {"type": {"value": "search"}}, {"type": {"value":"dashboard"}}, {"type": {"value":"visualization"}},{"type": {"value": "config"}},{"type": {"value": "url"}},{"type": {"value": "index-pattern"}} ] }}'
+
+

(3) This will result in a file containing the JSON-based definitions. Overwrite templates/kibana-index.json.

+

(4) After redeploying the code, your changes should now be a part of the default Metron dashboard.

+
+

Why do my dashboard components change their order when reloading the dashboard?

+

This has been a problem in Kibana 4.5.1 and perhaps other versions too. To address this problem find the definition for your dashboard in the Kibana index extract. It will look like the following.

+ +
+
+
{"_index":".kibana","_type":"dashboard","_id":"Metron-Dashboard",...
+
+

Extract the panelsJSON field from the dashboard definition. Reorder the definition of these panels so that they are ordered by row and column. The component in row 1 should come before the component in row 2, etc. After you have ordered the components in this way, Kibana will maintain the order of components in the dashboard.

+
+
+
+ +
+ +
+
+
Copyright © 2017. + All Rights Reserved. + +
+ + + +
+
+ + Added: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/monit/index.html ============================================================================== --- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/monit/index.html (added) +++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-deployment/roles/monit/index.html Fri Feb 10 20:21:39 2017 @@ -0,0 +1,327 @@ + + + + + + + + + Metron – Monit Integration + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Monit Integration

+

+

This role will leverage Monit as a process watchdog to manage sensors, topologies, and core services.

+ +
    + +
  • Monit can be used to start, stop, or check status of any of the sensors or topologies.
    • + +
  • When monitoring is enabled (on by default) if a process dies, it will be restarted.
    • + +
  • The Monit web interface is exposed at http://hostname:2812.
    • + +
  • The web interface username and password is defined by the monit_user and monit_password variables. These default to admin/monit.
    • + +
  • Monit CLI tools can also be used to simplify the process of managing Metron components.
    • + +
  • +

    The post-deployment report for Amazon-EC2 provides links to Monit’s web interface labeled as ‘Sensor Status’ and ‘Topology Status.’

    + +
    +
    +
      ok: [localhost] => {
+"Success": [
+    "Apache Metron deployed successfully",
+    "   Metron          @ http://ec2-52-39-143-62.us-west-2.compute.amazonaws.com:5000",
+    "   Ambari          @ http://ec2-52-39-4-93.us-west-2.compute.amazonaws.com:8080",
+    "   Sensor Status   @ http://ec2-52-39-4-93.us-west-2.compute.amazonaws.com:2812",
+    "   Topology Status @ http://ec2-52-39-130-62.us-west-2.compute.amazonaws.com:2812",
+    "For additional information, see https://metron.incubator.apache.org/'"
+]
+  }
+
    • +
+
+

Usage

+

Start all Metron components

+ +
+
+
monit start all
+
+

Stop all Metron components

+ +
+
+
monit stop all
+
+

Start an individual Metron component

+ +
+
+
monit start bro-parser
+
+

Start all components required to ingest Bro data

+ +
+
+
monit -g bro start
+
+

Start all parsers

+ +
+
+
monit -g parsers start
+
+

What is running?

+ +
+
+
monit summary
+
+
+
+
+ +
+ + + +