Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A8F09160C15 for ; Wed, 3 Jan 2018 15:53:43 +0100 (CET) Received: (qmail 40114 invoked by uid 500); 3 Jan 2018 14:53:42 -0000 Mailing-List: contact commits-help@metron.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.apache.org Delivered-To: mailing list commits@metron.apache.org Received: (qmail 40099 invoked by uid 99); 3 Jan 2018 14:53:42 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Jan 2018 14:53:42 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id DEFA2DFD7B; Wed, 3 Jan 2018 14:53:39 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: nickallen@apache.org To: commits@metron.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: metron git commit: METRON-1362 Improve Metron Deployment README (nickwallen) closes apache/metron#869 Date: Wed, 3 Jan 2018 14:53:39 +0000 (UTC) archived-at: Wed, 03 Jan 2018 14:53:44 -0000 Repository: metron Updated Branches: refs/heads/master 8a61b96b6 -> 01c26a77b METRON-1362 Improve Metron Deployment README (nickwallen) closes apache/metron#869 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/01c26a77 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/01c26a77 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/01c26a77 Branch: refs/heads/master Commit: 01c26a77b1041204b0bbbc544cc0a5d02e9339a8 Parents: 8a61b96 Author: nickwallen Authored: Wed Jan 3 09:52:57 2018 -0500 Committer: nickallen Committed: Wed Jan 3 09:52:57 2018 -0500 ---------------------------------------------------------------------- metron-deployment/README.md | 280 ++++++++----------- .../packaging/ambari/metron-mpack/README.md | 117 ++++++++ .../packaging/docker/ansible-docker/README.md | 56 ++-- .../packaging/docker/deb-docker/README.md | 40 +-- .../packaging/docker/rpm-docker/README.md | 57 +++- .../vagrant/full-dev-platform/README.md | 9 +- 6 files changed, 343 insertions(+), 216 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/01c26a77/metron-deployment/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/README.md b/metron-deployment/README.md index 0b9a6fe..86d04e7 100644 --- a/metron-deployment/README.md +++ b/metron-deployment/README.md @@ -15,178 +15,134 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -This set of playbooks can be used to deploy an Ambari-managed Hadoop cluster containing Metron services using Ansible. These playbooks target RHEL/CentOS 6.x operating -systems. -Installation consists of - -- Building Metron tarballs, RPMs and the Ambari MPack -- Deploying Ambari -- Leveraging Ambari to install: - * The required Hadoop Components - * Core Metron (Parsing, Enrichment, Indexing) - * Elasticsearch - * Kibana -- Starting All Services +This project contains tools for building, packaging, and deploying Apache Metron. Please refer to the following sections for more information on how to get Apache Metron running in your environment. -## Prerequisites -The following tools are required to run these scripts: - -- [Maven](https://maven.apache.org/) -- [Git](https://git-scm.com/) -- [Ansible](http://www.ansible.com/) (2.0.0.2 or 2.2.2.0) -- [Docker](https://www.docker.com/) (Docker for Mac on OSX) - -These scripts depend on two files for configuration: - -- hosts - declares which Ansible roles will be run on which hosts -- group_vars/all - various configuration settings needed to install Metron - -For production use, it is recommended that Metron be installed on an existing cluster managed by Ambari as described in the Installing Management Pack section below. -## Ambari -The Ambari playbook will install a Hadoop cluster including the Metron Services (Parsing, Enrichment, Indexing). Ambari will also install Elasticsearch and Kibana. - -Currently, the playbooks supports building a local development cluster running on one node or deploying to a 10 node cluster on AWS EC2. - -## Vagrant -There is a development environment based on Vagrant that is referred to as "Full Dev". This installs the entire Ambari/Metron stack. This is useful in testing out changes to the installation procedure. - -### Prerequsities -- Install [Vagrant](https://www.vagrantup.com/) (5.0.16+) -- Install the Hostmanager plugin for vagrant - Run `vagrant plugin install vagrant-hostmanager` on the machine where Vagrant is -installed - -### Full-Dev -Navigate to `metron/metron-deployment/vagrant/full-dev-platform` and run `vagrant up`. - -## Ambari Management Pack -An Ambari Management Pack can be built in order to make the Metron service available on top of an existing stack, rather than needing a direct stack update. - -This will set up -- Metron Parsers -- Enrichment -- Indexing -- GeoIP data -- Optional Elasticsearch -- Optional Kibana - -### Prerequisites -- A cluster managed by Ambari 2.4.2+ -- Metron RPMs available on the cluster in the /localrepo directory. See [RPMs](#rpms) for further information. -- [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the Management UI host - -### Building Management Pack -From `metron-deployment` run -``` -mvn clean package -``` - -A tar.gz that can be used with Ambari can be found at `metron-deployment/packaging/ambari/metron-mpack/target/` - -### Installing Management Pack -Before installing the mpack, update Storm's topology.classpath in Ambari to include '/etc/hbase/conf:/etc/hadoop/conf'. Restart Storm service. - -Place the mpack's tar.gz onto the node running Ambari Server. From the command line on this node, run -``` -ambari-server install-mpack --mpack= --verbose -``` - -This will make the services available in Ambari in the same manner as any services in a stack, e.g. through Add Services or during cluster install. -The Indexing / Parsers/ Enrichment masters should be colocated with a Kafka Broker (to create topics) and HBase client (to create the enrichment and theatintel tables). -This colocation is currently not enforced by Ambari, and should be managed by either a Service or Stack advisor as an enhancement. - -Several configuration parameters will need to be filled in, and should be pretty self explanatory (primarily a couple of Elasticsearch configs, and the Storm REST URL). Examples are provided in the descriptions on Ambari. -Notably, the URL for the GeoIP database that is preloaded (and is prefilled by default) can be set to use a `file:///` location - -After installation, a custom action is available in Ambari (where stop / start services are) to install Elasticsearch templates. Similar to this, a custom Kibana action to Load Template is available. - -Another custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing documentation](../metron-platform/metron-indexing) - -#### Offline installation -Currently there is only one point that would reach out to the internet during an install. This is the URL for the GeoIP database information. - -The RPMs DO NOT reach out to the internet (because there is currently no hosting for them). They look on the local filesystem in `/localrepo`. - -### Current Limitations -There are a set of limitations that should be addressed based to improve the current state of the mpacks. - -- There is currently no hosting for RPMs remotely. They will have to be built locally. -- Colocation of appropriate services should be enforced by Ambari. See [#Installing Management Pack] for more details. -- Storm's topology.classpath is not updated with the Metron service install and needs to be updated separately. -- Several configuration parameters used when installing the Metron service could (and should) be grabbed from Ambari. Install will require them to be manually entered. -- Need to handle upgrading Metron - -## RPMs -RPMs can be built to install the components in metron-platform. These RPMs are built in a Docker container and placed into `target`. - -Components in the RPMs: -- metron-common -- metron-data-management -- metron-elasticsearch -- metron-enrichment -- metron-parsers -- metron-pcap -- metron-solr -- stellar-common + * [How do I deploy Metron with Ambari?](#how-do-i-deploy-metron-with-ambari) + * [How do I deploy Metron on a single VM?](#how-do-i-deploy-metron-on-a-single-vm) + * [How do I build RPM packages?](#how-do-i-build-rpm-packages) + * [How do I build DEB packages?](#how-do-i-build-deb-packages) + * [How do I deploy Metron within AWS?](#how-do-i-deploy-metron-within-aws) + * [How do I build Metron with Docker?](#how-do-i-build-metron-with-docker) -### Prerequisites -- Docker. The image detailed in: `metron-deployment/packaging/docker/rpm-docker/README.md` will automatically be built (or rebuilt if necessary). -- Artifacts for metron-platform have been produced. E.g. `mvn clean package -DskipTests` in `metron-platform` -The artifacts are required because there is a dependency on modules not expressed via Maven (we grab the resulting assemblies, but don't need the jars). These are -- metron-common -- metron-data-management -- metron-elasticsearch -- metron-enrichment -- metron-indexing -- metron-parsers -- metron-pcap-backend -- metron-solr -- metron-profiler -- metron-config +How do I deploy Metron with Ambari? +----------------------------------- -### Building RPMs -``` -cd metron-deployment -mvn clean package -Pbuild-rpms -``` - -The output RPM files will land in `target/RPMS/noarch`. They can be installed with the standard -``` -rpm -i -``` +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. -## Kibana Dashboards - -The dashboards installed by the Kibana custom action are managed by the dashboard.p file. This file is created by exporting existing dashboards from a running Kibana instance. - -To create a new version of the file, make any necessary changes to Kibana (e.g. on full-dev), and export with the appropriate script. - -``` -python packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboardindex.py \ -$ES_HOST 9200 \ -packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard/dashboard.p -s -``` +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. -Build the Ambari Mpack to get the dashboard updated appropriately. +#### What is this good for? -Once the MPack is installed, run the Kibana service's action "Load Template" to install dashboards. This will completely overwrite the .kibana in Elasticsearch, so use with caution. +* If you want to see how Metron can really scale by deploying it on your own hardware, or even in the cloud, this is the best option for you. -## Kerberos -The MPack can allow Metron to be installed and then Kerberized, or installed on top of an already Kerberized cluster. This is done through Ambari's standard Kerberization setup. +* If you want to run a proof-of-concept to see how Apache Metron can benefit your organization, then this is the way to do it. -### Caveats -* For nodes using a Metron client and a local repo, the repo must exist on all nodes (e.g via createrepo). This repo can be empty; only the main Metron services need the RPMs. -* A Metron client must be installed on each supervisor node in a secured cluster. This is to ensure that the Metron keytab and client_jaas.conf get distributed in order to allow reading and writing from Kafka. - * When Metron is already installed on the cluster, this should be done before Kerberizing. - * When addding Metron to an already Kerberized cluster, ensure that all supervisor nodes receive a Metron client. -* Storm (and Metron) must be restarted after Metron is installed on an already Kerberized cluster. Several Storm configs get updated, and Metron will be unable to write to Kafka without a restart. - * Kerberizing a cluster with an existing Metron already has restarts of all services during Kerberization, so it's unneeded. +#### How? -Instructions for setup on Full Dev can be found at [Kerberos-ambari-setup.md](Kerberos-ambari-setup.md). These instructions reference the manual install instructions. +To deploy Apache Metron using Ambari, follow the instructions at [packaging/ambari/metron-mpack](packaging/ambari/metron-mpack). -### Kerberos Without an MPack -Using the MPack is preferred, but instructions for Kerberizing manually can be found at [Kerberos-manual-setup.md](Kerberos-manual-setup.md). These instructions are reference by the Ambari Kerberos install instructions and include commands for setting up a KDC. -## TODO -- Support Ubuntu deployments +How do I deploy Metron on a single VM? +-------------------------------------- + +This will deploy Metron and all of its dependencies on a virtual machine running on your computer. + +#### What is this good for? + +* If you are new to Metron and want to explore the functionality that it offers, this is good place to start. + +* If you are a developer contributing to the Apache Metron project, this is also a great way to test your changes. + +#### What is this **not** good for? + +* This VM is **not** intended for processing anything beyond the most basic, low volume work loads. + +* Additional services should **not** be installed along side Metron in this VM. + +* This VM should **not** be used to run a proof-of-concept for Apache Metron within your organization. + +Running Metron within the resource constraints of a single VM is incredibly challenging. Failing to respect this warning, will cause various services to fail mysteriously as the system runs into memory and processing limits. + +#### How? + +To deploy Metron in a VM running on your computer, follow the instructions at [vagrant/full-dev-platform](vagrant/full-dev-platform) + + +How do I build RPM packages? +---------------------------- + +This provides RPM packages that allow you to install Metron on an RPM-based operating system like CentOS. + +#### What is this good for? + +* If you want to manually install Apache Metron on an RPM-based system like CentOS, installation can be simplified by using these packages. + +* If you want a guided installation process using Ambari on an RPM-based system, then these RPMs are a necessary prerequisite. + +#### What is this **not** good for? + +* If you want a complete, guided installation process, use Ambari rather than just these packages. Installing Metron using **only** these RPMs still leaves a considerable amount of configuration necessary to get Metron running. Installing with Ambari automates these additional steps. + +#### How? + +To build the RPM packages, follow the instructions at [packaging/docker/rpm-docker](packaging/docker/rpm-docker). + + +How do I build DEB packages? +------------------------------- + +This builds installable DEB packages that allow you to install Metron on an APT-based operating system like Ubuntu. + +#### What is this good for? + +* If you want to manually install Metron on a APT-based system like Ubuntu, installation can be simplified by using these packages. + +* If you want a guided installation process using Ambari on an APT-based system, then these DEBs are a necessary prerequisite. + +#### What is this **not** good for? + +* If you want a complete, guided installation process, use Ambari rather than just these packages. Installing Metron using **only** these RPMs still leaves a considerable amount of configuration necessary to get Metron running. Installing with Ambari automates these additional steps. + +#### How? + +To build the DEB packages, follow the instructions at [packaging/docker/deb-docker](packaging/docker/deb-docker). + + +How do I deploy Metron within AWS? +---------------------------------- + +This deploys Apache Metron on an automatically provisioned 10-node cluster running in Amazon Web Service's EC2 platform. + +This installs real sources of telemetry like Bro, Snort, and YAF, but feeds those sensors with canned pcap data. + +#### What is this good for? + +* If you are a Metron developer wanting to test at-scale on a multi-node cluster, then this is the right option for you. + +#### What is this **not** good for? + +* If you want to run Metron in AWS with real data for either testing or production, then this is NOT the right option for you. + +* **WARNING** This is only intended for creating an ephemeral cluster for brief periods of testing. This deployment method has the following severe limitations. + * The cluster is not secured in any way. It is up to you to manually secure it. + * The cluster will not survive a reboot. + +#### How? + +Follow the instructions available at [amazon-ec2](amazon-ec2). + + +How do I build Metron with Docker? +---------------------------------- + +This provides a Docker containing all of the prerequisites required to build Metron. This allows you to easily build Metron without installing all of the build dependencies manually. + +#### What is this good for? + +* If you want to build Metron, but do not want to manually install all of the build dependencies, then this is a good option. + +#### How? + +Follow the instructions available at [packaging/docker/ansible-docker](packaging/docker/ansible-docker). http://git-wip-us.apache.org/repos/asf/metron/blob/01c26a77/metron-deployment/packaging/ambari/metron-mpack/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/ambari/metron-mpack/README.md b/metron-deployment/packaging/ambari/metron-mpack/README.md new file mode 100644 index 0000000..17fc4f6 --- /dev/null +++ b/metron-deployment/packaging/ambari/metron-mpack/README.md @@ -0,0 +1,117 @@ + + +This provides a Management Pack (MPack) extension for [Apache Ambari](https://ambari.apache.org/) that simplifies the provisioning, management and monitoring of Metron on clusters of any size. + +This allows you to easily install Metron using a simple, guided process. This also allows you to monitor cluster health and even secure your cluster with kerberos. + +### Prerequisites + +* Ambari 2.4.2+ + +* Installable Metron packages (either RPMs or DEBs) located in a repository on each host at `/localrepo`. + +* A [Node.js](https://nodejs.org/en/download/package-manager/) repository installed on the host running the Management and Alarm UI. + +### Quick Start + +1. Build the Metron MPack. Execute the following command from the project's root directory. + ``` + mvn clean package -Pmpack -DskipTests + ``` + +1. This results in the Mpack being produced at the following location. + ``` + metron-deployment/packaging/ambari/metron-mpack/target/metron_mpack-x.y.z.0.tar.gz + ``` + +1. Copy the tarball to the host where Ambari Server is installed. + +1. Ensure that Ambari Server is stopped. + +1. Install the MPack. + ``` + ambari-server install-mpack --mpack=metron_mpack-x.y.z.0.tar.gz --verbose + ``` + +1. Install the Metron packages (RPMs or DEBs) in a local repository on each host where a Metron component is installed. By default, the repository is expected to exist at `/localrepo`. + + On hosts where only a Metron client is installed, the local repository must exist, but it does not need to contain Metron packages. For example to create an empty repository for an RPM-based system, run the following commands. + + ``` + yum install createrepo + mkdir /localrepo + cd /localrepo + createrepo + ``` + +1. Metron will now be available as an installable service within Ambari. + +### Installation Notes + +The MPack will make all Metron services available in Ambari in the same manner as any other services in a stack. These can be installed using Ambari's user interface using "Add Services" or during an initial cluster install. + +#### Kerberization + +The MPack allows Metron to be automatically kerberized in two different ways. +* Metron can be installed on a non-kerberized cluster and then the entire cluster can be kerberized using Ambari. +* Alternatively, Metron can be installed on top of an already kerberized cluster. + +Using the MPack is preferred, but instructions for manually Kerberizing a cluster with Metron can be found at [Kerberos-manual-setup.md](../../Kerberos-manual-setup.md). + +##### Metron Client + +A "Metron Client" must be installed on each supervisor node in a kerberized cluster. This client ensures that the Metron keytab and `client_jaas.conf` get distributed to each node in order to allow reading and writing from Kafka. +* When Metron is already installed on the cluster, installation of the "Metron Client" should be done before Kerberizing. +* When adding Metron to an already Kerberized cluster, ensure that all supervisor nodes receive a Metron client. + +##### Restarts + +Storm (and the Metron topologies) must be restarted after Metron is installed on an already Kerberized cluster. The restart triggers several Storm configurations to get updated and Metron will be unable to write to Kafka without a restart. + +Kerberizing a cluster with a pre-existing Metron, automatically restarts all services during Kerberization. No additional manual restart is needed in this case. + +#### Zeppelin Import + +A custom action is available in Ambari to import Zeppelin dashboards. See the [metron-indexing documentation](../../../../metron-platform/metron-indexing) for more information. + +#### Kibana Dashboards + +The dashboards installed by the Kibana custom action are managed by the `dashboard.p` file. This file is created by exporting existing dashboards from a running Kibana instance. + +To create a new version of the file, make any necessary changes to Kibana and run the following commands to export your changes. + ``` + cd packaging/ambari/metron-mpack/src/main/resources/common-services/KIBANA/4.5.1/package/scripts/dashboard + python dashboardindex.py $ES_HOST 9200 dashboard.p -s + ``` + +#### Offline Installation + +Retrieval of the GeoIP database is the only point during installation that reaches out to the internet. For an offline installation, the URL for the GeoIP database can be manually set to a local path on the file system such as `file:///home/root/geoip/GeoLite2-City.mmdb.gz`. + +The RPMs DO NOT reach out to the internet (because there is currently no hosting for them). They look on the local filesystem in `/localrepo`. + +#### Limitations + +There are a few limitations that should be addressed to improve the Metron MPack installation. + +* There is no external hosting for Metron packages (either RPMs or DEBs). These have to be built locally and installed on each host in a repository located at `/localrepo`. + +* Several configuration parameters used when installing Metron could retrieved from Ambari rather than requiring user input. + +* The MPack does not support upgrades. http://git-wip-us.apache.org/repos/asf/metron/blob/01c26a77/metron-deployment/packaging/docker/ansible-docker/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/ansible-docker/README.md b/metron-deployment/packaging/docker/ansible-docker/README.md index 910446b..d7466b6 100644 --- a/metron-deployment/packaging/docker/ansible-docker/README.md +++ b/metron-deployment/packaging/docker/ansible-docker/README.md @@ -15,42 +15,42 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -The Metron ansible-docker container is provided in an effort reduce the installation burden of building Metron. -It may also be used to deploy Metron in a live environment. -It is provisioned with software required to sucessfully build metron and run the deployment scripts. -## Building the Container -1. Install Docker ( https://www.docker.com/products/overview ) -2. Navigate to \/metron-deployment/packaging/docker/ansible-docker -3. Build the container `docker build -t ansible-docker:2.0.0.2 .` +This provides a Docker Container containing all of the prerequisites required to build Metron. This allows you to easily build Metron without installing all of the build dependencies manually. -## Using the Container to build metron -anytime after building the container you can run it with the following command +### Prerequisites -`docker run -it -v \:/root/metron ansible-docker:2.0.0.2 bash` +* Docker: Please ensure that Docker is installed and that the daemon is running. -If you are going to build metron multiple times, you may want to map the /root/.m2 maven -repo from outside of the container so that you don't start with an empty repo every build and have to download -the world. +### Quick Start -`docker run -it -v \:/root/metron -v \:/root/.m2 ansible-docker:2.0.0.2 bash` +1. Build the Docker container. + ``` + cd metron-deployment/packaging/docker/ansible-docker + docker build -t ansible-docker:latest . + ``` -After running the container: +1. Launch the container. + ``` + docker run -it \ + -v `pwd`/../../../..:/root/metron \ + -v ~/.m2:/root/.m2 \ + ansible-docker:latest bash + ``` -1. cd /root/metron -2. run build commands, for example: - - build metron without tests : `mvn clean package -DskipTests` - - build metron and build the rpms as well : `mvn clean install && cd metron-deployment && mvn package -P build-rpms` + This maps the Metron source code along with your local Maven repository to the container. This will prevent you from having to re-download all of these dependencies each time you build Metron. -If you wish to use this build with a vagrant instance, then after building with rpms as above, modify -your usual vagrant up command to skip the build role, as so: - -`vagrant --ansible-skip-tags="build" up` +1. Execute the following inside the Docker container to build Metron. + ``` + cd /root/metron + mvn clean package -DskipTests + ``` -## Using the Container for deployment +### Notes -> Note these instructions are outdated - -Full instructions are found on the wiki at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65144361 +If you wish to use this build with a vagrant instance, then after building with rpms as above, modify +your usual vagrant up command to skip the build role, as so: + ``` + vagrant --ansible-skip-tags="build" up + ``` http://git-wip-us.apache.org/repos/asf/metron/blob/01c26a77/metron-deployment/packaging/docker/deb-docker/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/deb-docker/README.md b/metron-deployment/packaging/docker/deb-docker/README.md index 69e1aa6..a0ac0b3 100644 --- a/metron-deployment/packaging/docker/deb-docker/README.md +++ b/metron-deployment/packaging/docker/deb-docker/README.md @@ -15,45 +15,51 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -This project builds DEB packages for installing Apache Metron on Ubuntu. -**WARNING**: The DEB packages are a recent addition to Metron. These packages have not undergone the same level of testing as the RPM packages. Improvements and more rigerous testing of these packages is underway and will improve in future releases. Until then, use these at your own risk. +This project builds packages that allow you to install Apache Metron on an APT-based operating system like Ubuntu. + +If you are installing Metron using Ambari, these packages are necessary prerequisites when installing on an APT-based platform like Ubuntu. Installing Metron using **only** these packages still leaves a considerable amount of configuration necessary to get Metron running. Installing with Ambari automates these additional steps. + +**WARNING**: These packages are a recent addition to Metron. These have not undergone the same level of testing as the RPM packages. Improvements and more rigorous testing of these packages is underway and will improve in future releases. Until then, use these at your own risk. ### Prerequisites -Building these packages requires Docker. Please ensure that Docker is installed and that the daemon is running. +* Docker: Please ensure that Docker is installed and that the daemon is running. -### How do I create packages for Ubuntu? +### Quick Start -Running the following from Metron's top-level source directory will build Metron along with the DEB packages. Using the `package` or `install` target are both sufficient. +1. Execute the following command from the project's root directory. + ``` + mvn clean package -DskipTests -Pbuild-debs + ``` - ``` - mvn clean package -DskipTests -Pbuild-debs - ``` +1. The packages will be accessible from the following location once the build process completes. + ``` + metron-deployment/packaging/docker/deb-docker/target + ``` -If Metron has already been built, just the DEB packages can be built by doing the following. +### Build Packages - ``` - cd metron-deployment - mvn clean package -Pbuild-debs - ``` +If Metron has already been built, just the DEB packages can be built by executing the following commands. + ``` + cd metron-deployment + mvn clean package -Pbuild-debs + ``` -### How does this really work? +### How does this work? Using the `build-debs` profile as shown above effectively automates the following steps. 1. Copy the tarball for each Metron sub-project to the `target` working directory. 1. Build a Docker image of a Ubuntu Trusty host called `docker-deb` that contains all of the tools needed to build the packages. - ``` docker build -t deb-docker . ``` 1. Execute the `build.sh` script within a Docker container. The argument passed to the build script is the current version of Metron. - ``` - docker run -v `pwd`:/root deb-docker:latest /bin/bash -c ./build.sh 0.4.3 + docker run -v `pwd`:/root deb-docker:latest /bin/bash -c ./build.sh ``` 1. This results in the DEBs being generated within the `target` directory. http://git-wip-us.apache.org/repos/asf/metron/blob/01c26a77/metron-deployment/packaging/docker/rpm-docker/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/packaging/docker/rpm-docker/README.md b/metron-deployment/packaging/docker/rpm-docker/README.md index e9fe2e6..7667ed1 100644 --- a/metron-deployment/packaging/docker/rpm-docker/README.md +++ b/metron-deployment/packaging/docker/rpm-docker/README.md @@ -15,11 +15,52 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> -# Overview -The Metron ansible-docker container is provided in an effort reduce the installation burden of deploying Metron in a live envirionment. -It is provisioned with software required to sucessfully run the deployment scripts. - -## Building the Container -1. Install Docker ( https://www.docker.com/products/overview ) -2. Navigate to \/metron-deployment/packaging/rpm-docker -3. Build the container `docker build -t rpm-docker .` + +This project provides RPM packages that allow you to install Metron on an RPM-based operating system like CentOS. + +If you are installing Metron using Ambari, these packages are necessary prerequisites when installing on an RPM-based platform like CentOS. Installing Metron using **only** these packages still leaves a considerable amount of configuration necessary to get Metron running. Installing with Ambari automates these additional steps. + +### Prerequisites + +* Docker: Please ensure that Docker is installed and that the daemon is running. + +### Quick Start + +1. Execute the following command from the project's root directory. + ``` + mvn clean package -DskipTests -Pbuild-rpms + ``` + +1. The packages will be accessible from the following location once the build process completes. + ``` + metron-deployment/packaging/docker/rpm-docker/RPMS/noarch + ``` + +### Build Packages + +If Metron has already been built, just the RPM packages can be built by executing the following commands. + ``` + cd metron-deployment + mvn clean package -Pbuild-debs + ``` + +### How does this work? + +Using the `build-rpms` profile as shown above, effectively automates the following steps. + +1. Copy the tarball for each Metron sub-project to the `target` working directory. + +1. Build a Docker image of a CentOS host called `rpm-docker` that contains all of the tools needed to build the packages. + ``` + docker build -t rpm-docker . + ``` + +1. Execute the `build.sh` script within the Docker container. The argument passed to the build script is the current version of Metron. + ``` + docker run -v `pwd`:/root rpm-docker:latest /bin/bash -c ./build.sh + ``` + +1. This results in the RPMs being generated within the following directory. + ``` + metron-deployment/packaging/docker/rpm-docker/RPMS/noarch + ``` http://git-wip-us.apache.org/repos/asf/metron/blob/01c26a77/metron-deployment/vagrant/full-dev-platform/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/vagrant/full-dev-platform/README.md b/metron-deployment/vagrant/full-dev-platform/README.md index 3da383c..4a58915 100644 --- a/metron-deployment/vagrant/full-dev-platform/README.md +++ b/metron-deployment/vagrant/full-dev-platform/README.md @@ -36,8 +36,15 @@ The computer used to deploy Apache Metron will need to have the following compon - [Virtualbox](https://virtualbox.org) 5.0+ - Python 2.7 - Maven 3.3.9 + - C++11 compliant compiler, like [GCC](https://gcc.gnu.org/projects/cxx-status.html#cxx11) -#### macOS +Running the following script can help validate whether you have all the prerequisites installed and running correctly. + + ``` + metron-deployment/scripts/platform-info.sh + ``` + +#### How do I install these on MacOS? Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project.