; Wed, 5 Jul 2017 06:56:45 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r20299 [8/18] - in /release/metron/0.4.0: ./ site-book/ site-book/css/ site-book/images/ site-book/images/logos/ site-book/images/profiles/ site-book/img/ site-book/js/ site-book/metron-analytics/ site-book/metron-analytics/metron-maas-serv... Date: Wed, 05 Jul 2017 06:56:42 -0000 To: commits@metron.apache.org From: cestella@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20170705065645.E868E3A519B@svn01-us-west.apache.org> archived-at: Wed, 05 Jul 2017 06:56:52 -0000 Added: release/metron/0.4.0/site-book/metron-deployment/amazon-ec2/index.html ============================================================================== --- release/metron/0.4.0/site-book/metron-deployment/amazon-ec2/index.html (added) +++ release/metron/0.4.0/site-book/metron-deployment/amazon-ec2/index.html Wed Jul 5 06:56:42 2017 @@ -0,0 +1,561 @@ + + + + + + + + + Metron – Apache Metron on Amazon EC2 + + + + + + + + + + + + + + + + + +

+ + +

+ +

+ + +

User Documentation
+ + + + Metron +
- + + + + Upgrading +
- + + + + Analytics +
- + + + + Deployment +
  - + + + + Kerberos-ambari-setup +
  - + + + + Kerberos-manual-setup +
  - + + Amazon-ec2 +
  - + + + + Ansible-docker +
  - + + + + Rpm-docker +
  - + + + + Packer-build +
  - + + + + Roles +
  - + + + + Vagrant +
  +
- + + + + Docker +
- + + + + Config +
- + + + + Rest +
- + + + + Platform +
- + + + + Sensors +
+

+ + + +

+ +

+ + +

+ +

Apache Metron on Amazon EC2

This project fully automates the provisioning of Apache Metron on Amazon EC2 infrastructure. Starting with only your Amazon EC2 credentials, this project will create a fully-functioning, end-to-end, multi-node cluster running Apache Metron.

Warning: Amazon will charge for the use of their resources when running Apache Metron. The amount will vary based on the number and size of hosts, along with current Amazon pricing structure. Be sure to stop or terminate all of the hosts instantiated by Apache Metron when not in use to avoid unnecessary charges.

Getting Started

Prerequisites

The host used to deploy Apache Metron will need the following software tools installed. The following versions are known to work as of the time of this writing, but by no means are these the only working versions.

+ +

Ansible 2.0.0.2 or 2.2.2.0
Python 2.7.11
Maven 3.3.9

Any platform that supports these tools is suitable, but the following instructions cover only macOS. The easiest means of installing these tools on a Mac is to use the excellent Homebrew project.

+ +

+
Install Homebrew by running the following command in a terminal. Refer to the Homebrew home page for the latest installation instructions.
+ +
+
+
```
  /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
+
```
+
With Homebrew installed, run the following command in a terminal to install all of the required tools.
+ +
+
+
```
  brew cask install java
+  brew install maven git
+
```
+
Install Ansible by following the instructions here.

Ensure that a public SSH key is located at ~/.ssh/id_rsa.pub.

+ +

  $ cat ~/.ssh/id_rsa.pub
+  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQChv5GJxPjR39UJV7VY17ivbLVlxFrH7UHwh1Jsjem4d1eYiAtde5N2y65/HRNxWbhYli9ED8k0/MRP92ejewucEbrPNq5mytPqdC4IvZ98Ln2GbqTDwvlP3T7xa/wYFOpFsOmXXql8216wSrnrS4f3XK7ze34S6/VmY+lsBYnr3dzyj8sG/mexpJgFS/w83mWJV0e/ryf4Hd7P6DZ5fO+nmTXfKNK22ga4ctcnbZ+toYcPL+ODCh8598XCKVo97XjwF5OxN3vl1p1HHguo3cHB4H1OIaqX5mUt59gFIZcAXUME89PO6NUiZDd3RTstpf125nQVkQAHu2fvW96/f037 nick@localhost
+

If this file does not exist, run the following command at a terminal and accept all defaults. Only the public key, not the private key, will be uploaded to Amazon and configured on each host to enable SSH connectivity. While it is possible to create and use an alternative key those details will not be covered.

+ +

  ssh-keygen -t rsa
+

Amazon Web Services

If you already have an Amazon Web Services account that you have used to deploy EC2 hosts, then you should be able to skip the next few steps.

+ +

+
Head over to Amazon Web Services and create an account. As part of the account creation process you will need to provide a credit card to cover any charges that may apply.
+
Create a set of user credentials through Amazon’s Identity and Access Management (IAM) dashboard. On the IAM dashboard menu click “Users” and then “Create New User”. Provide a name and ensure that “Generate an access key for each user” remains checked. Download the credentials and keep them for later use.
+
While still in Amazon’s Identity and Access Management (IAM) dashboard, click on the user that was previously created. Click the “Permissions” tab and then the “Attach Policy” button. Attach the following policies to the user.
+ +
- AmazonEC2FullAccess
- AmazonVPCFullAccess
+
Apache Metron uses the official, open source CentOS 6 Amazon Machine Image (AMI). If you have never used this AMI before then you will need to accept Amazon’s terms and conditions. Navigate to the web page for this AMI and click the “Continue” button. Choose the “Manual Launch” tab then click the “Accept Software Terms” button.

Having successfully created your Amazon Web Services account, hopefully you will find that the most difficult tasks are behind us.

Deploy Metron

+ +

+
Use the Amazon access key by exporting its values via the shell’s environment. This allows Ansible to authenticate with Amazon EC2. For example:
+ +
+
+
```
  export AWS_ACCESS_KEY_ID="AKIAI6NRFEO27E5FFELQ"
+  export AWS_SECRET_ACCESS_KEY="vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed"
+
```
+
Notice: You must replace the access key values above with values from your own access key.
+
Start the Apache Metron deployment process. When prompted provide a unique name for your Metron environment or accept the default.
+ +
+
+
```
  $ ./run.sh
+  Metron Environment [metron-test]: my-metron-env
+  ...
+
```
+
The process is likely to take between 70-90 minutes. Fortunately, everything is fully automated and you should feel free to grab a coffee.

Explore Metron

+ +

After the deployment has completed successfully, a message like the following will be displayed. Navigate to the specified resources to explore your newly minted Apache Metron environment.

+ +

  TASK [debug] *******************************************************************
+  ok: [localhost] => {
+  "Success": [
+      "Apache Metron deployed successfully",
+      "   Metron  @  http://ec2-52-37-255-142.us-west-2.compute.amazonaws.com:5000",
+      "   Ambari  @  http://ec2-52-37-225-202.us-west-2.compute.amazonaws.com:8080",
+      "   Sensors @  ec2-52-37-225-202.us-west-2.compute.amazonaws.com on tap0",
+      "For additional information, see https://metron.apache.org/'"
+  ]
+  }
+

+
Each of the provisioned hosts will be accessible from the internet. Connecting to one over SSH as the user centos will not require a password as it will authenticate with the pre-defined SSH key.
+ +
+
+
```
  ssh centos@ec2-52-91-215-174.compute-1.amazonaws.com
+
```

Advanced Usage

Multiple Environments

This process can support provisioning of multiple, isolated environments. Simply change the env settings in conf/defaults.yml. For example, you might provision separate development, test, and production environments.

+ +

env: metron-test
+

Selective Provisioning

To provision only subsets of the entire Metron deployment, Ansible tags can be specified. For example, to only deploy the sensors on an Amazon EC2 environment, run the following command.

+ +

ansible-playbook -i ec2.py playbook.yml --tags "ec2,sensors"
+

Custom SSH Key

By default, the playbook will attempt to register your public SSH key ~/.ssh/id_rsa.pub with each provisioned host. This enables Ansible to communicate with each host using an SSH connection. If would prefer to use another key simply add the path to the public key file to the key_file property in conf/defaults.yml.

For example, generate a new SSH key for Metron that will be stored at ~/.ssh/my-metron-key.

+ +

$ ssh-keygen -q -f ~/.ssh/my-metron-key
+Enter passphrase (empty for no passphrase):
+Enter same passphrase again:
+

Add the path to the newly created SSH public key to conf/defaults.yml.

+ +

key_file: ~/.ssh/metron-private-key.pub
+

Common Errors

Error: [unsupported_operation_exception] custom format isn’t supported

This error might be seen within Metron’s default dashboard in Kibana 4. This occurs when the index templates do not exist for the Snort, Bro or YAF indices in Elasticsearch.

The dashboard expects fields to be of a certain type. If the index templates have not been loaded correctly, the data types for the fields in these indices will be incorrect and the dashboard will display this error.

Solution

If you see this error, please report your findings by creating a JIRA or dropping an email to the Metron Users mailing list. Follow these steps to work around the problem.

+ +

+
Define which Elasticsearch host to interact with. Any Elasticsearch host should work.
+ +
+
+
```
export ES_HOST="http://ec2-52-25-237-20.us-west-2.compute.amazonaws.com:9200"
+
```
+
Confirm the index templates are in fact missing.
+ +
+
+
```
curl -s -XGET $ES_HOST/_template
+
```

Manually load the index templates.

+ +

cd metron-deployment
+curl -s -XPOST $ES_HOST/_template/bro_index -d @roles/metron_elasticsearch_templates/files/es_templates/bro_index.template
+curl -s -XPOST $ES_HOST/_template/snort_index -d @roles/metron_elasticsearch_templates/files/es_templates/snort_index.template
+curl -s -XPOST $ES_HOST/_template/yaf_index -d @roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template
+

Delete the existing indexes. Only a new index will use the templates defined in the previous step.

+ +

curl -s -XDELETE "$ES_HOST/yaf_index*"
+curl -s -XDELETE "$ES_HOST/bro_index*"
+curl -s -XDELETE "$ES_HOST/snort_index*"
+

+
Open up Kibana and wait for the new indexes to be created. The dashboard should now work.

Error: ‘No handler was ready to authenticate…Check your credentials’

+ +

TASK [Define keypair] **********************************************************
+failed: [localhost] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPP
+  r9jEFrn0yun3PYNidJ/...david@hasselhoff.com) => {"failed": true, "item": "ssh-r
+  sa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXbcb1AlWsEPPr9jEFr... david@hasselhoff.com",
+  "msg": "No handler was ready to authenticate. 1 handlers were checked.
+  ['HmacAuthV4Handler'] Check your credentials"}
+

Solution 1

This occurs when Ansible does not have the correct AWS access keys. The following commands must return a valid access key that is defined within Amazon’s Identity and Access Management console.

+ +

$ echo $AWS_ACCESS_KEY_ID
+AKIAI6NRFEO27E5FFELQ
+
+$ echo $AWS_SECRET_ACCESS_KEY
+vTDydWJQnAer7OWauUS150i+9Np7hfCXrrVVP6ed
+

Solution 2

This error can occur if you have exported the correct AWS access key, but you are using sudo to run the Ansible playbook. Do not use the sudo command when running the Ansible playbook.

Error: ‘OptInRequired: … you need to accept terms and subscribe’

+ +

TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
+fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
+"Instance creation failed => OptInRequired: In order to use this AWS Marketplace
+product you need to accept terms and subscribe. To do so please visit
+http://aws.amazon.com/marketplace/pp?sku=6x5jmcajty9edm3f211pqjfn2"}
+to retry, use: --limit @playbook.retry
+

Solution

Apache Metron uses the official CentOS 6 Amazon Machine Image when provisioning hosts. Amazon requires that you accept certain terms and conditions when using any Amazon Machine Image (AMI). Follow the link provided in the error message to accept the terms and conditions then re-run the playbook.

Error: ‘PendingVerification: Your account is currently being verified’

+ +

TASK [metron-test: Instantiate 1 host(s) as sensors,ambari_master,metron,ec2] **
+fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
+"Instance creation failed => PendingVerification: Your account is currently
+being verified. Verification normally takes less than 2 hours. Until your
+account is verified, you may not be able to launch additional instances or
+create additional volumes. If you are still receiving this message after more
+than 2 hours, please let us know by writing to aws-verification@amazon.com. We
+appreciate your patience."}
+to retry, use: --limit @playbook.retry
+

Solution

This will occur if you are attempting to deploy Apache Metron using a newly created Amazon Web Services account. Follow the advice of the message and wait until Amazon’s verification process is complete. Amazon has some additional advice for dealing with this error and more.

+ +

+
Your account is pending verification. Until the verification process is complete, you may not be able to carry out requests with this account. If you have questions, contact AWS Support.
+

Error: ‘Instance creation failed => InstanceLimitExceeded’

+ +

TASK [metron-test: Instantiate 3 host(s) as search,metron,ec2] *****************
+fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg":
+"Instance creation failed => InstanceLimitExceeded: You have requested more
+instances (11) than your current instance limit of 10 allows for the specified
+instance type. Please visit http://aws.amazon.com/contact-us/ec2-request to
+request an adjustment to this limit."}
+to retry, use: --limit @playbook.retry
+

Solution

This will occur if Apache Metron attempts to deploy more host instances than allowed by your account. The total number of instances required for Apache Metron can be reduced by editing deployment/amazon-ec/playbook.yml. Perhaps a better alternative is to request of Amazon that this limit be increased. Amazon has some additional advice for dealing with this error and more.

+ +

+
You’ve reached the limit on the number of instances you can run concurrently. The limit depends on the instance type. For more information, see How many instances can I run in Amazon EC2. If you need additional instances, complete the Amazon EC2 Instance Request Form.
+

Error: ‘SSH encountered an unknown error during the connection’

+ +

TASK [setup] *******************************************************************
+fatal: [ec2-52-26-113-221.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {
+  "changed": false, "msg": "SSH encountered an unknown error during the
+  connection. We recommend you re-run the command using -vvvv, which will enable
+  SSH debugging output to help diagnose the issue", "unreachable": true}
+

Solution

This most often indicates that Ansible cannot connect to the host with the SSH key that it has access to. This could occur if hosts are provisioned with one SSH key, but the playbook is executed subsequently with a different SSH key. The issue can be addressed by either altering the key_file variable to point to the key that was used to provision the hosts or by simply terminating all hosts and re-running the playbook.

+ +

+ + + + Added: release/metron/0.4.0/site-book/metron-deployment/index.html ============================================================================== --- release/metron/0.4.0/site-book/metron-deployment/index.html (added) +++ release/metron/0.4.0/site-book/metron-deployment/index.html Wed Jul 5 06:56:42 2017 @@ -0,0 +1,499 @@ + + + + + + + + + Metron – Overview + + + + + + + + + + + + + + + + + +

+ + +

+ +

+ + +

User Documentation
+ + + + Metron +
- + + + + Upgrading +
- + + + + Analytics +
- + + Deployment +
  - + + + + Kerberos-ambari-setup +
  - + + + + Kerberos-manual-setup +
  - + + + + Amazon-ec2 +
  - + + + + Ansible-docker +
  - + + + + Rpm-docker +
  - + + + + Packer-build +
  - + + + + Roles +
  - + + + + Vagrant +
  +
- + + + + Docker +
- + + + + Config +
- + + + + Rest +
- + + + + Platform +
- + + + + Sensors +
+

+ + + +

+ +

+ + +

+ +

Overview

This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating systems.

Installation consists of -

+ +

Building Metron tarballs, RPMs and the Ambari MPack
Deploying Ambari
Leveraging Ambari to install: + +
- The required Hadoop Components
- Core Metron (Parsing, Enrichment, Indexing)
- Elasticsearch
- Kibana
Starting All Services

Prerequisites

The following tools are required to run these scripts:

+ +

Maven
Git
Ansible (2.0.0.2 or 2.2.2.0)
Docker (Docker for Mac on OSX)

These scripts depend on two files for configuration:

+ +

hosts - declares which Ansible roles will be run on which hosts
group_vars/all - various configuration settings needed to install Metron

For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below.

Ambari

The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana.

Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2.

Vagrant

There are current two Vagrant modes, full-dev and quick-dev. Full-dev installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. Quick-dev re-installs the core Metron Services (Parsing, Enrichment, and Indexing)on a pre-built instance. Use quick-dev for testing out changes to core Metron services.

Prerequsities

+ +

Install Vagrant (5.0.16+)
Install the Hostmanager plugin for vagrant - Run vagrant plugin install vagrant-hostmanager on the machine where Vagrant is installed

Full-Dev

Navigate to metron/metron-deployment/vagrant/full-dev-platform and run vagrant up.

Quick-Dev

Navigate to metron/metron-deployment/vagrant/quick-dev-platform and run vagrant up.

Ambari Management Pack

An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update.

This will set up

+ +

Metron Parsers
Enrichment
Indexing
GeoIP data
Optional Elasticsearch
Optional Kibana

Prerequisites

+ +

A cluster managed by Ambari 2.4.2+
Metron RPMs available on the cluster in the /localrepo directory. See RPM for further information.
Node.js repository installed on the Management UI host

Building Management Pack

From metron-deployment run

+ +

mvn clean package
+

A tar.gz that can be used with Ambari can be found at metron-deployment/packaging/ambari/metron-mpack/target/

Installing Management Pack

Before installing the mpack, update Storm’s topology.classpath in Ambari to include ‘/etc/hbase/conf:/etc/hadoop/conf’. Restart Storm service.

Place the mpack’s tar.gz onto the node running Ambari Server. From the command line on this node, run

+ +

ambari-server install-mpack --mpack=<mpack_location> --verbose
+

This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement.

Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a file:/// location

After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearch templates. Similar to this, a custom Kibana action to Load Template is available.

Another custom action is available in Ambari to import Zeppelin dashboards. See the metron-indexing documentation

Offline installation

Currently there is only one point that would reach out to the internet during an install. This is the URL for the GeoIP database information.

The RPMs DO NOT reach out to the internet (because there is currently no hosting for them). They look on the local filesystem in /localrepo.

Current Limitations

There are a set of limitations that should be addressed based to improve the current state of the mpacks.

+ +

There is currently no hosting for RPMs remotely. They will have to be built locally.
Colocation of appropriate services should be enforced by Ambari. See [#Installing Management Pack] for more details.
Storm’s topology.classpath is not updated with the Metron service install and needs to be updated separately.
Several configuration parameters used when installing the Metron service could (and should) be grabbed from Ambari. Install will require them to be manually entered.
Need to handle upgrading Metron

RPM

RPMs can be built to install the components in metron-platform. These RPMs are built in a Docker container and placed into target.

Components in the RPMs:

+ +

metron-common
metron-data-management
metron-elasticsearch
metron-enrichment
metron-parsers
metron-pcap
metron-solr

Prerequisites

+ +

Docker. The image detailed in: metron-deployment/packaging/docker/rpm-docker/README.md will automatically be built (or rebuilt if necessary).
Artifacts for metron-platform have been produced. E.g. mvn clean package -DskipTests in metron-platform

The artifacts are required because there is a dependency on modules not expressed via Maven (we grab the resulting assemblies, but don’t need the jars). These are

+ +

metron-common
metron-data-management
metron-elasticsearch
metron-enrichment
metron-indexing
metron-parsers
metron-pcap-backend
metron-solr
metron-profiler
metron-config

Building RPMs

From metron-deployment run

+ +

mvn clean package -Pbuild-rpms
+

The output RPM files will land in target/RPMS/noarch. They can be installed with the standard

+ +

rpm -i <package>
+

Kibana Dashboards

The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance.

To create a new version of the file, make any necessary changes to Kibana (e.g. on quick-dev), and export with the appropriate script.

+ +

python packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboardindex.py \
+$ES_HOST 9200 \
+packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboard.p -s
+

Build the Ambari Mpack to get the dashboard updated appropriately.

Once the MPack is installed, run the Kibana service’s action “Load Template” to install dashboards. This will completely overwrite the .kibana in Elasticsearch, so use with caution.

Kerberos

The MPack can allow Metron to be installed and then Kerberized, or installed on top of an already Kerberized cluster. This is done through Ambari’s standard Kerberization setup.

Caveats

+ +

For nodes using a Metron client and a local repo, the repo must exist on all nodes (e.g via createrepo). This repo can be empty; only the main Metron services need the RPMs.
A Metron client must be installed on each supervisor node in a secured cluster. This is to ensure that the Metron keytab and client_jaas.conf get distributed in order to allow reading and writing from Kafka. + +
- When Metron is already installed on the cluster, this should be done before Kerberizing.
- When addding Metron to an already Kerberized cluster, ensure that all supervisor nodes receive a Metron client.
Storm (and Metron) must be restarted after Metron is installed on an already Kerberized cluster. Several Storm configs get updated, and Metron will be unable to write to Kafka without a restart. + +
- Kerberizing a cluster with an existing Metron already has restarts of all services during Kerberization, so it’s unneeded.

Instructions for setup on Full Dev can be found at Kerberos-ambari-setup.md. These instructions reference the manual install instructions.

Kerberos Without an MPack

Using the MPack is preferred, but instructions for Kerberizing manually can be found at Kerberos-manual-setup.md. These instructions are reference by the Ambari Kerberos install instructions and include commands for setting up a KDC.

TODO

+ +

Support Ubuntu deployments

+ +

+ + + + Added: release/metron/0.4.0/site-book/metron-deployment/packaging/docker/ansible-docker/index.html ============================================================================== --- release/metron/0.4.0/site-book/metron-deployment/packaging/docker/ansible-docker/index.html (added) +++ release/metron/0.4.0/site-book/metron-deployment/packaging/docker/ansible-docker/index.html Wed Jul 5 06:56:42 2017 @@ -0,0 +1,265 @@ + + + + + + + + + Metron – Overview + + + + + + + + + + + + + + + + + +

+ + +

+ +

+ + +

User Documentation
+ + + + Metron +
- + + + + Upgrading +
- + + + + Analytics +
- + + + + Deployment +
  - + + + + Kerberos-ambari-setup +
  - + + + + Kerberos-manual-setup +
  - + + + + Amazon-ec2 +
  - + + Ansible-docker +
  - + + + + Rpm-docker +
  - + + + + Packer-build +
  - + + + + Roles +
  - + + + + Vagrant +
  +
- + + + + Docker +
- + + + + Config +
- + + + + Rest +
- + + + + Platform +
- + + + + Sensors +
+

+ + + +

+ +

+ + +

+ +

Overview

The Metron ansible-docker container is provided in an effort reduce the installation burden of deploying Metron in a live envirionment. It is provisioned with software required to sucessfully run the deployment scripts.

Building the Container

+ +

Install Docker ( https://www.docker.com/products/overview )
Navigate to <project-directory>/metron-deployment/packaging/docker/ansible-docker
Build the container docker build -t ansible-docker:2.0.0.2 .

Using the Container

Full instructions are found on the wiki at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361

tl;dr:

+ +

docker run -it -v <project-directory>:/root/metron ansible-docker:2.0.0.2 bash
cd /root/metron
mvn clean package -DskipTests

+ +

+ + + + Added: release/metron/0.4.0/site-book/metron-deployment/packaging/docker/rpm-docker/index.html ============================================================================== --- release/metron/0.4.0/site-book/metron-deployment/packaging/docker/rpm-docker/index.html (added) +++ release/metron/0.4.0/site-book/metron-deployment/packaging/docker/rpm-docker/index.html Wed Jul 5 06:56:42 2017 @@ -0,0 +1,252 @@ + + + + + + + + + Metron – Overview + + + + + + + + + + + + + + + + + +

+ + +

+ +

+ + +

User Documentation
+ + + + Metron +
- + + + + Upgrading +
- + + + + Analytics +
- + + + + Deployment +
  - + + + + Kerberos-ambari-setup +
  - + + + + Kerberos-manual-setup +
  - + + + + Amazon-ec2 +
  - + + + + Ansible-docker +
  - + + Rpm-docker +
  - + + + + Packer-build +
  - + + + + Roles +
  - + + + + Vagrant +
  +
- + + + + Docker +
- + + + + Config +
- + + + + Rest +
- + + + + Platform +
- + + + + Sensors +
+

+ + + +

+ +

+ + +

+ +

Overview

Building the Container

+ +

Install Docker ( https://www.docker.com/products/overview )
Navigate to <project-directory>/metron-deployment/packaging/rpm-docker
Build the container docker build -t rpm-docker .

+ +

+ + + + Added: release/metron/0.4.0/site-book/metron-deployment/packaging/packer-build/index.html ============================================================================== --- release/metron/0.4.0/site-book/metron-deployment/packaging/packer-build/index.html (added) +++ release/metron/0.4.0/site-book/metron-deployment/packaging/packer-build/index.html Wed Jul 5 06:56:42 2017 @@ -0,0 +1,298 @@ + + + + + + + + + Metron – Build Metron Images + + + + + + + + + + + + + + + + + +

+ + +

+ +

+ + +

User Documentation
+ + + + Metron +
- + + + + Upgrading +
- + + + + Analytics +
- + + + + Deployment +
  - + + + + Kerberos-ambari-setup +
  - + + + + Kerberos-manual-setup +
  - + + + + Amazon-ec2 +
  - + + + + Ansible-docker +
  - + + + + Rpm-docker +
  - + + Packer-build +
  - + + + + Roles +
  - + + + + Vagrant +
  +
- + + + + Docker +
- + + + + Config +
- + + + + Rest +
- + + + + Platform +
- + + + + Sensors +
+

+ + + +

+ +

+ + +

+ +

Build Metron Images

Based on the fantastic Bento project developed by Chef.

Images Provided

+ +

base-centos-6.7: Centos 6.7 + HDP. Used in the full-dev-platform Vagrant image
quick-dev-centos-6.7: Centos 6.7 + HDP + Metron. Used for the quick-dev-platform Vagrant image.

Prerequisites

+ +

Packer 0.12.2
Virtualbox 5.0.16+ (Tested with 5.0.20)

Build Both Images ———————- Navigate to <your-project-directory>/metron-deployment/packer-build Execute bin/bento build

Packer will build both images and export .box files to the ./builds directory.

Build Single Images ———————- Navigate to your-project-directory/metron-deployment/packer-build

+ +

Base Centos (full-dev)

+ +

bin/bento build base-centos-6.7.json
+

+ +

Quick Dev

+ +

bin/bento build quick-dev-centos-6.7.json
+

Using Your New Box File ———————- Modify the relevant Vagrantfile (full-dev-platform or quick-dev-platform) replacing the lines:

+ +

<pre><code>config.vm.box = "<i>box_name</i>"
+config.ssh.insert_key = true</code></pre>
+

with

+ +

<pre></code>config.vm.box = "<i>test_box_name</i>"
+config.vm.box = "<i>PathToBoxfile/Boxfilename</i>"
+config.ssh.insert_key = true</code></pre>
+

Launch the image as usual.

Node: Vagrant will cache boxes, you can force Vagrant to reload your box by running vagrant box remove test_box_name before launching your new image.

+ +

+ + + +