From commits-return-2590-archive-asf-public=cust-asf.ponee.io@metron.apache.org Fri Jan 26 21:56:22 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 6AE5618079E for ; Fri, 26 Jan 2018 21:56:21 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 59653160C3E; Fri, 26 Jan 2018 20:56:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DA4F4160C56 for ; Fri, 26 Jan 2018 21:56:18 +0100 (CET) Received: (qmail 47575 invoked by uid 500); 26 Jan 2018 20:56:18 -0000 Mailing-List: contact commits-help@metron.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.apache.org Delivered-To: mailing list commits@metron.apache.org Received: (qmail 47510 invoked by uid 99); 26 Jan 2018 20:56:18 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Jan 2018 20:56:18 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 3A189F3518; Fri, 26 Jan 2018 20:56:15 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: cestella@apache.org To: commits@metron.apache.org Date: Fri, 26 Jan 2018 20:56:33 -0000 Message-Id: <7bf22f5f7a974908939c57282fc7c277@git.apache.org> In-Reply-To: <71e201ba159d460c9cfa728b71a6f717@git.apache.org> References: <71e201ba159d460c9cfa728b71a6f717@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [19/19] metron git commit: METRON-1370 Create Full Dev Equivalent for Ubuntu (nickwallen via cestella) closes apache/incubator-metron#903 METRON-1370 Create Full Dev Equivalent for Ubuntu (nickwallen via cestella) closes apache/incubator-metron#903 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/6f267991 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/6f267991 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/6f267991 Branch: refs/heads/master Commit: 6f2679919a9e6c9f3a146d52b10c6dbe2ce55180 Parents: 878003a Author: nickwallen Authored: Fri Jan 26 15:56:01 2018 -0500 Committer: cstella Committed: Fri Jan 26 15:56:01 2018 -0500 ---------------------------------------------------------------------- README.md | 32 +- dev-utilities/release-utils/metron-rc-check | 2 +- metron-contrib/metron-docker/README.md | 14 +- metron-deployment/Kerberos-manual-setup.md | 4 +- metron-deployment/README.md | 2 +- metron-deployment/amazon-ec2/ansible.cfg | 4 +- metron-deployment/amazon-ec2/conf/defaults.yml | 8 - metron-deployment/amazon-ec2/playbook.yml | 2 +- metron-deployment/ansible/README.md | 24 + .../extra_modules/ambari_cluster_state.py | 395 + .../extra_modules/ambari_service_state.py | 352 + .../ansible/playbooks/ambari_install.yml | 54 + .../ansible/playbooks/docker_probe_install.yml | 62 + .../ansible/playbooks/metron_build.yml | 23 + .../ansible/playbooks/metron_full_install.yml | 46 + .../ansible/playbooks/metron_install.yml | 75 + .../ansible/playbooks/sensor_install.yml | 59 + metron-deployment/ansible/roles/README.md | 24 + .../roles/ambari_common/defaults/main.yml | 23 + .../ansible/roles/ambari_common/meta/main.yml | 21 + .../ambari_common/tasks/ambari-repo-centos.yml | 22 + .../ambari_common/tasks/ambari-repo-ubuntu.yml | 23 + .../ambari_common/tasks/elasticsearch-repo.yml | 40 + .../roles/ambari_common/tasks/hostname.yml | 32 + .../ambari_common/tasks/iptables-centos.yml | 20 + .../ambari_common/tasks/iptables-ubuntu.yml | 19 + .../roles/ambari_common/tasks/logrotate.yml | 22 + .../ansible/roles/ambari_common/tasks/main.yml | 39 + .../roles/ambari_common/tasks/nodejs.yml | 34 + .../templates/metron-hadoop-logrotate.yml | 149 + .../roles/ambari_config/defaults/main.yml | 38 + .../ansible/roles/ambari_config/meta/main.yml | 22 + .../roles/ambari_config/tasks/dependencies.yml | 24 + .../ansible/roles/ambari_config/tasks/main.yml | 39 + .../roles/ambari_config/vars/single_node_vm.yml | 135 + .../roles/ambari_config/vars/small_cluster.yml | 149 + .../roles/ambari_gather_facts/meta/main.yml | 21 + .../roles/ambari_gather_facts/tasks/main.yml | 234 + .../roles/ambari_master/defaults/main.yml | 21 + .../roles/ambari_master/tasks/ambari.yml | 36 + .../ansible/roles/ambari_master/tasks/main.yml | 24 + .../ansible/roles/ambari_master/tasks/mpack.yml | 26 + .../roles/ambari_slave/defaults/main.yml | 18 + .../roles/ambari_slave/files/hostname.sh | 19 + .../ansible/roles/ambari_slave/meta/main.yml | 19 + .../roles/ambari_slave/tasks/install-agent.yml | 24 + .../ansible/roles/ambari_slave/tasks/main.yml | 46 + .../ansible/roles/bro/meta/main.yml | 22 + .../ansible/roles/bro/tasks/bro.yml | 59 + .../ansible/roles/bro/tasks/dependencies.yml | 48 + .../ansible/roles/bro/tasks/librdkafka.yml | 39 + .../ansible/roles/bro/tasks/main.yml | 22 + .../roles/bro/tasks/metron-bro-plugin-kafka.yml | 49 + .../ansible/roles/bro/tasks/nic.yml | 19 + .../ansible/roles/bro/vars/main.yml | 34 + .../ansible/roles/build-tools/meta/main.yml | 19 + .../ansible/roles/build-tools/tasks/main.yml | 34 + .../roles/deployment-report/defaults/main.yml | 19 + .../roles/deployment-report/meta/main.yml | 19 + .../roles/deployment-report/tasks/main.yml | 32 + .../ansible/roles/enable-swap/defaults/main.yml | 19 + .../roles/enable-swap/tasks/enable-swap.yml | 35 + .../ansible/roles/enable-swap/tasks/main.yml | 31 + .../ansible/roles/epel/tasks/main.yml | 20 + .../ansible/roles/fastcapa/defaults/main.yml | 42 + .../ansible/roles/fastcapa/meta/main.yml | 19 + .../ansible/roles/fastcapa/tasks/debug.yml | 26 + .../roles/fastcapa/tasks/dependencies.yml | 39 + .../ansible/roles/fastcapa/tasks/dpdk.yml | 59 + .../ansible/roles/fastcapa/tasks/fastcapa.yml | 46 + .../ansible/roles/fastcapa/tasks/kernel.yml | 51 + .../ansible/roles/fastcapa/tasks/main.yml | 22 + .../ansible/roles/fastcapa/templates/fastcapa | 145 + .../roles/fastcapa/templates/fastcapa.conf | 67 + .../ansible/roles/httplib2/tasks/main.yml | 20 + .../ansible/roles/java_jdk/defaults/main.yml | 18 + .../ansible/roles/java_jdk/tasks/main.yml | 34 + .../roles/kafka-broker/defaults/main.yml | 18 + .../ansible/roles/kafka-broker/meta/main.yml | 18 + .../ansible/roles/kafka-broker/tasks/main.yml | 41 + .../roles/kafka-client/defaults/main.yml | 18 + .../ansible/roles/kafka-client/tasks/main.yml | 30 + .../ansible/roles/librdkafka/defaults/main.yml | 20 + .../roles/librdkafka/tasks/dependencies.yml | 40 + .../roles/librdkafka/tasks/librdkafka.yml | 39 + .../ansible/roles/librdkafka/tasks/main.yml | 19 + .../roles/libselinux-python/tasks/main.yml | 26 + .../roles/load_web_templates/meta/main.yml | 19 + .../roles/load_web_templates/tasks/main.yml | 24 + .../roles/metron-builder/defaults/main.yml | 18 + .../roles/metron-builder/tasks/build-debs.yml | 26 + .../roles/metron-builder/tasks/build-rpms.yml | 26 + .../roles/metron-builder/tasks/build.yml | 26 + .../roles/metron-builder/tasks/clean.yml | 32 + .../ansible/roles/metron-builder/tasks/main.yml | 35 + .../roles/metron-packages/defaults/main.yml | 21 + .../tasks/copy_packages_centos.yml | 25 + .../tasks/copy_packages_ubuntu.yml | 31 + .../metron-packages/tasks/create_directory.yml | 27 + .../tasks/create_repo_centos.yml | 24 + .../tasks/create_repo_ubuntu.yml | 32 + .../roles/metron-packages/tasks/main.yml | 52 + .../roles/metron-packages/tasks/validate.yml | 24 + .../files/es_templates/error_index.template | 57 + .../roles/metron_pcapservice/defaults/main.yml | 28 + .../roles/metron_pcapservice/meta/main.yml | 19 + .../metron_pcapservice/tasks/config-hbase.yml | 26 + .../roles/metron_pcapservice/tasks/main.yml | 25 + .../metron_pcapservice/tasks/pcapservice.yml | 30 + .../templates/metron-pcapservice-logrotate.yml | 27 + .../metron_pcapservice/templates/pcapservice | 84 + .../ansible/roles/ntp/tasks/main.yml | 37 + .../ansible/roles/opentaxii/README.md | 178 + .../ansible/roles/opentaxii/defaults/main.yml | 43 + .../ansible/roles/opentaxii/meta/main.yml | 17 + .../roles/opentaxii/tasks/dependencies.yml | 37 + .../roles/opentaxii/tasks/hailataxii.yml | 45 + .../ansible/roles/opentaxii/tasks/main.yml | 20 + .../ansible/roles/opentaxii/tasks/opentaxii.yml | 50 + .../opentaxii/templates/collection-status.py | 32 + .../roles/opentaxii/templates/collections.yml | 19 + .../ansible/roles/opentaxii/templates/opentaxii | 176 + .../opentaxii/templates/opentaxii-conf.yml | 38 + .../roles/opentaxii/templates/services.yml | 69 + .../ansible/roles/pcap_replay/README.md | 61 + .../ansible/roles/pcap_replay/defaults/main.yml | 21 + .../ansible/roles/pcap_replay/meta/main.yml | 17 + .../roles/pcap_replay/tasks/dependencies.yml | 32 + .../ansible/roles/pcap_replay/tasks/main.yml | 20 + .../ansible/roles/pcap_replay/tasks/service.yml | 25 + .../roles/pcap_replay/tasks/tcpreplay.yml | 38 + .../roles/pcap_replay/templates/pcap-replay | 97 + .../ansible/roles/pycapa/defaults/main.yml | 26 + .../ansible/roles/pycapa/meta/main.yml | 20 + .../ansible/roles/pycapa/tasks/dependencies.yml | 36 + .../ansible/roles/pycapa/tasks/main.yml | 21 + .../roles/pycapa/tasks/pycapa-service.yml | 22 + .../ansible/roles/pycapa/tasks/pycapa.yml | 40 + .../ansible/roles/pycapa/templates/pycapa | 115 + .../ansible/roles/python-pip/tasks/main.yml | 24 + .../ansible/roles/sensor-stubs/README.md | 99 + .../roles/sensor-stubs/defaults/main.yml | 25 + .../ansible/roles/sensor-stubs/files/bro.out | 1346 + .../ansible/roles/sensor-stubs/files/snort.out | 27404 +++++++++++++++++ .../ansible/roles/sensor-stubs/files/yaf.out | 22164 +++++++++++++ .../ansible/roles/sensor-stubs/tasks/main.yml | 43 + .../roles/sensor-stubs/templates/sensor-stubs | 154 + .../roles/sensor-stubs/templates/start-bro-stub | 55 + .../sensor-stubs/templates/start-snort-stub | 55 + .../roles/sensor-stubs/templates/start-yaf-stub | 56 + .../ansible/roles/sensor-test-mode/README.md | 44 + .../roles/sensor-test-mode/defaults/main.yml | 20 + .../roles/sensor-test-mode/files/example.pcap | Bin 0 -> 507865 bytes .../roles/sensor-test-mode/meta/main.yml | 19 + .../roles/sensor-test-mode/tasks/main.yml | 25 + .../roles/sensor-test-mode/tasks/pcap.yml | 22 + .../roles/sensor-test-mode/tasks/snort.yml | 36 + .../roles/sensor-test-mode/tasks/yaf.yml | 30 + .../ansible/roles/snort/defaults/main.yml | 28 + .../ansible/roles/snort/files/snort.conf | 730 + .../ansible/roles/snort/meta/main.yml | 23 + .../ansible/roles/snort/tasks/daq.yml | 36 + .../ansible/roles/snort/tasks/main.yml | 21 + .../ansible/roles/snort/tasks/nic.yml | 19 + .../ansible/roles/snort/tasks/producer.yml | 31 + .../ansible/roles/snort/tasks/snort.yml | 85 + .../roles/snort/templates/snort-producer | 83 + .../snort/templates/start-snort-producer.sh | 24 + .../roles/tap_interface/defaults/main.yml | 19 + .../ansible/roles/tap_interface/tasks/main.yml | 35 + .../ansible/roles/yaf/defaults/main.yml | 29 + .../ansible/roles/yaf/meta/main.yml | 22 + .../ansible/roles/yaf/tasks/fixbuf.yml | 37 + .../ansible/roles/yaf/tasks/main.yml | 20 + .../ansible/roles/yaf/tasks/nic.yml | 19 + .../ansible/roles/yaf/tasks/yaf.yml | 53 + .../ansible/roles/yaf/templates/start-yaf.sh | 25 + .../ansible/roles/yaf/templates/yaf | 83 + metron-deployment/development/README.md | 24 + .../development/centos6/.gitignore | 1 + metron-deployment/development/centos6/README.md | 105 + .../development/centos6/Vagrantfile | 92 + .../development/centos6/ansible.cfg | 27 + .../centos6/ansible/inventory/group_vars/all | 85 + .../development/centos6/ansible/inventory/hosts | 47 + .../development/fastcapa/.gitignore | 1 + .../development/fastcapa/README.md | 138 + .../development/fastcapa/centos-7.1/Vagrantfile | 72 + .../development/fastcapa/centos-7.1/ansible.cfg | 22 + .../fastcapa/centos-7.1/vars/main.yml | 52 + .../development/fastcapa/centos-7.4/Vagrantfile | 72 + .../development/fastcapa/centos-7.4/ansible.cfg | 22 + .../fastcapa/centos-7.4/vars/main.yml | 52 + .../development/fastcapa/playbook.yml | 49 + .../tasks/validate-messages-received.yml | 38 + .../fastcapa/tasks/validate-packets-sent.yml | 41 + .../development/ubuntu14/README.md | 106 + .../development/ubuntu14/Vagrantfile | 87 + .../development/ubuntu14/ansible.cfg | 27 + .../ubuntu14/ansible/inventory/group_vars/all | 85 + .../ubuntu14/ansible/inventory/hosts | 47 + .../development/ubuntu14/ansible/playbook.yml | 25 + .../extra_modules/ambari_cluster_state.py | 395 - .../extra_modules/ambari_service_state.py | 352 - .../inventory/full-dev-platform/group_vars/all | 89 - .../inventory/full-dev-platform/hosts | 47 - .../METRON/CURRENT/repos/repoinfo.xml | 8 +- .../5.6.2/configuration/elastic-sysconfig.xml | 7 +- .../5.6.2/configuration/elastic-systemd.xml | 30 + .../ELASTICSEARCH/5.6.2/metainfo.xml | 19 +- .../5.6.2/package/scripts/elastic.py | 86 - .../5.6.2/package/scripts/elastic_commands.py | 253 + .../5.6.2/package/scripts/elastic_master.py | 49 +- .../5.6.2/package/scripts/elastic_slave.py | 55 +- .../5.6.2/package/scripts/params.py | 13 + .../5.6.2/package/scripts/service_check.py | 80 +- .../5.6.2/package/scripts/slave.py | 80 - .../5.6.2/package/scripts/status_params.py | 1 + .../common-services/KIBANA/5.6.2/metainfo.xml | 21 +- .../KIBANA/5.6.2/package/scripts/common.py | 56 + .../5.6.2/package/scripts/kibana_master.py | 56 +- .../CURRENT/package/scripts/indexing_master.py | 3 +- .../packaging/docker/deb-docker/build.sh | 59 +- .../docker/deb-docker/debian/changelog | 22 - .../packaging/docker/deb-docker/debian/control | 28 - .../docker/deb-docker/debian/copyright | 35 - .../deb-docker/debian/metron-alerts/postinst | 26 + .../deb-docker/debian/metron-alerts/postrm | 25 + .../deb-docker/debian/metron-alerts/preinst | 29 + .../deb-docker/debian/metron-alerts/prerm | 30 + .../deb-docker/debian/metron-config/postinst | 26 + .../deb-docker/debian/metron-config/postrm | 25 + .../deb-docker/debian/metron-config/preinst | 29 + .../deb-docker/debian/metron-config/prerm | 30 + .../docker/deb-docker/debian/metron/changelog | 28 + .../docker/deb-docker/debian/metron/control | 34 + .../docker/deb-docker/debian/metron/copyright | 41 + .../packaging/docker/deb-docker/pom.xml | 6 + .../packaging/docker/rpm-docker/pom.xml | 6 + metron-deployment/playbooks/ambari_install.yml | 61 - .../playbooks/docker_probe_install.yml | 62 - metron-deployment/playbooks/metron_build.yml | 23 - .../playbooks/metron_full_install.yml | 34 - metron-deployment/playbooks/metron_install.yml | 102 - metron-deployment/roles/README.md | 24 - .../roles/ambari_common/defaults/main.yml | 20 - .../roles/ambari_common/meta/main.yml | 21 - .../roles/ambari_common/tasks/main.yml | 54 - .../templates/metron-hadoop-logrotate.yml | 149 - .../roles/ambari_config/defaults/main.yml | 38 - .../roles/ambari_config/meta/main.yml | 22 - .../roles/ambari_config/tasks/main.yml | 46 - .../roles/ambari_config/vars/single_node_vm.yml | 135 - .../roles/ambari_config/vars/small_cluster.yml | 149 - .../roles/ambari_gather_facts/meta/main.yml | 21 - .../roles/ambari_gather_facts/tasks/main.yml | 234 - .../roles/ambari_master/defaults/main.yml | 19 - .../roles/ambari_master/tasks/main.yml | 52 - .../roles/ambari_slave/defaults/main.yml | 18 - .../roles/ambari_slave/files/hostname.sh | 19 - .../roles/ambari_slave/meta/main.yml | 19 - .../roles/ambari_slave/tasks/main.yml | 56 - metron-deployment/roles/bro/meta/main.yml | 22 - metron-deployment/roles/bro/tasks/bro.yml | 59 - .../roles/bro/tasks/dependencies.yml | 48 - .../roles/bro/tasks/librdkafka.yml | 39 - metron-deployment/roles/bro/tasks/main.yml | 22 - .../roles/bro/tasks/metron-bro-plugin-kafka.yml | 49 - metron-deployment/roles/bro/tasks/nic.yml | 19 - metron-deployment/roles/bro/vars/main.yml | 34 - .../roles/build-tools/meta/main.yml | 19 - .../roles/build-tools/tasks/main.yml | 34 - .../roles/deployment-report/defaults/main.yml | 18 - .../roles/deployment-report/meta/main.yml | 19 - .../roles/deployment-report/tasks/main.yml | 37 - metron-deployment/roles/epel/tasks/main.yml | 19 - .../roles/fastcapa/defaults/main.yml | 42 - metron-deployment/roles/fastcapa/meta/main.yml | 19 - .../roles/fastcapa/tasks/debug.yml | 26 - .../roles/fastcapa/tasks/dependencies.yml | 39 - metron-deployment/roles/fastcapa/tasks/dpdk.yml | 59 - .../roles/fastcapa/tasks/fastcapa.yml | 46 - .../roles/fastcapa/tasks/kernel.yml | 51 - metron-deployment/roles/fastcapa/tasks/main.yml | 22 - .../roles/fastcapa/templates/fastcapa | 145 - .../roles/fastcapa/templates/fastcapa.conf | 67 - metron-deployment/roles/httplib2/tasks/main.yml | 20 - .../roles/java_jdk/defaults/main.yml | 18 - metron-deployment/roles/java_jdk/tasks/main.yml | 34 - .../roles/kafka-broker/defaults/main.yml | 18 - .../roles/kafka-broker/meta/main.yml | 18 - .../roles/kafka-broker/tasks/main.yml | 41 - .../roles/kafka-client/defaults/main.yml | 18 - .../roles/kafka-client/tasks/main.yml | 30 - .../roles/librdkafka/defaults/main.yml | 20 - .../roles/librdkafka/tasks/dependencies.yml | 40 - .../roles/librdkafka/tasks/librdkafka.yml | 39 - .../roles/librdkafka/tasks/main.yml | 19 - .../roles/libselinux-python/tasks/main.yml | 25 - .../roles/load_web_templates/meta/main.yml | 19 - .../roles/load_web_templates/tasks/main.yml | 24 - .../roles/metron-builder/defaults/main.yml | 18 - .../roles/metron-builder/tasks/main.yml | 26 - .../roles/metron-rpms/defaults/main.yml | 18 - .../roles/metron-rpms/tasks/copy_rpms.yml | 26 - .../metron-rpms/tasks/create_directory.yml | 28 - .../roles/metron-rpms/tasks/create_repo.yml | 28 - .../roles/metron-rpms/tasks/main.yml | 21 - .../files/es_templates/error_index.template | 57 - .../roles/metron_pcapservice/defaults/main.yml | 29 - .../roles/metron_pcapservice/meta/main.yml | 19 - .../metron_pcapservice/tasks/config-hbase.yml | 26 - .../roles/metron_pcapservice/tasks/main.yml | 25 - .../metron_pcapservice/tasks/pcapservice.yml | 30 - .../templates/metron-pcapservice-logrotate.yml | 27 - .../metron_pcapservice/templates/pcapservice | 84 - .../roles/metron_streaming/defaults/main.yml | 98 - .../roles/metron_streaming/files/extractor.json | 12 - .../roles/metron_streaming/meta/main.yml | 21 - .../metron_streaming/tasks/copy_bundles.yml | 52 - .../roles/metron_streaming/tasks/es_purge.yml | 42 - .../roles/metron_streaming/tasks/geoip.yml | 22 - .../metron_streaming/tasks/grok_upload.yml | 37 - .../metron_streaming/tasks/hdfs_filesystem.yml | 56 - .../roles/metron_streaming/tasks/hdfs_purge.yml | 52 - .../roles/metron_streaming/tasks/main.yml | 53 - .../metron_streaming/tasks/source_config.yml | 39 - .../metron_streaming/tasks/threat_intel.yml | 46 - .../roles/metron_streaming/tasks/topologies.yml | 86 - .../templates/config/elasticsearch.global.json | 7 - .../templates/config/solr.global.json | 6 - .../metron_streaming/templates/threat_ip.csv | 37 - .../roles/monit-start/defaults/main.yml | 23 - .../roles/monit-start/tasks/main.yml | 53 - metron-deployment/roles/monit/README.md | 79 - metron-deployment/roles/monit/defaults/main.yml | 24 - metron-deployment/roles/monit/tasks/main.yml | 28 - .../roles/monit/tasks/monit-definitions.yml | 22 - .../monit/tasks/monit-sensor-definitions.yml | 38 - .../monit/tasks/monit-stub-definitions.yml | 30 - metron-deployment/roles/monit/tasks/monit.yml | 27 - .../roles/monit/templates/monit/bro-stub.monit | 25 - .../roles/monit/templates/monit/bro.monit | 25 - .../roles/monit/templates/monit/monit.conf | 29 - .../monit/templates/monit/pcap-replay.monit | 24 - .../roles/monit/templates/monit/pycapa.monit | 24 - .../monit/templates/monit/snort-stub.monit | 25 - .../roles/monit/templates/monit/snort.monit | 33 - .../roles/monit/templates/monit/yaf-stub.monit | 25 - .../roles/monit/templates/monit/yaf.monit | 24 - metron-deployment/roles/ntp/tasks/main.yml | 31 - metron-deployment/roles/opentaxii/README.md | 178 - .../roles/opentaxii/defaults/main.yml | 43 - metron-deployment/roles/opentaxii/meta/main.yml | 17 - .../roles/opentaxii/tasks/dependencies.yml | 37 - .../roles/opentaxii/tasks/hailataxii.yml | 45 - .../roles/opentaxii/tasks/main.yml | 20 - .../roles/opentaxii/tasks/opentaxii.yml | 50 - .../opentaxii/templates/collection-status.py | 32 - .../roles/opentaxii/templates/collections.yml | 19 - .../roles/opentaxii/templates/opentaxii | 176 - .../opentaxii/templates/opentaxii-conf.yml | 38 - .../roles/opentaxii/templates/services.yml | 69 - metron-deployment/roles/pcap_replay/README.md | 61 - .../roles/pcap_replay/defaults/main.yml | 21 - .../roles/pcap_replay/meta/main.yml | 17 - .../roles/pcap_replay/tasks/dependencies.yml | 32 - .../roles/pcap_replay/tasks/main.yml | 20 - .../roles/pcap_replay/tasks/service.yml | 22 - .../roles/pcap_replay/tasks/tcpreplay.yml | 38 - .../roles/pcap_replay/templates/pcap-replay | 97 - .../roles/pycapa/defaults/main.yml | 26 - metron-deployment/roles/pycapa/meta/main.yml | 20 - .../roles/pycapa/tasks/dependencies.yml | 36 - metron-deployment/roles/pycapa/tasks/main.yml | 21 - .../roles/pycapa/tasks/pycapa-service.yml | 22 - metron-deployment/roles/pycapa/tasks/pycapa.yml | 40 - metron-deployment/roles/pycapa/templates/pycapa | 115 - .../roles/python-pip/tasks/main.yml | 25 - metron-deployment/roles/sensor-stubs/README.md | 99 - .../roles/sensor-stubs/defaults/main.yml | 25 - .../roles/sensor-stubs/files/bro.out | 1346 - .../roles/sensor-stubs/files/snort.out | 27404 ----------------- .../roles/sensor-stubs/files/yaf.out | 22164 ------------- .../roles/sensor-stubs/tasks/main.yml | 40 - .../roles/sensor-stubs/templates/sensor-stubs | 154 - .../roles/sensor-stubs/templates/start-bro-stub | 55 - .../sensor-stubs/templates/start-snort-stub | 55 - .../roles/sensor-stubs/templates/start-yaf-stub | 56 - .../roles/sensor-test-mode/README.md | 44 - .../roles/sensor-test-mode/defaults/main.yml | 20 - .../roles/sensor-test-mode/files/example.pcap | Bin 507865 -> 0 bytes .../roles/sensor-test-mode/meta/main.yml | 19 - .../roles/sensor-test-mode/tasks/main.yml | 25 - .../roles/sensor-test-mode/tasks/pcap.yml | 22 - .../roles/sensor-test-mode/tasks/snort.yml | 36 - .../roles/sensor-test-mode/tasks/yaf.yml | 30 - metron-deployment/roles/snort/defaults/main.yml | 28 - metron-deployment/roles/snort/files/snort.conf | 730 - metron-deployment/roles/snort/meta/main.yml | 23 - metron-deployment/roles/snort/tasks/daq.yml | 36 - metron-deployment/roles/snort/tasks/main.yml | 21 - metron-deployment/roles/snort/tasks/nic.yml | 19 - .../roles/snort/tasks/producer.yml | 28 - metron-deployment/roles/snort/tasks/snort.yml | 85 - .../roles/snort/templates/snort-producer | 83 - .../snort/templates/start-snort-producer.sh | 24 - .../roles/tap_interface/defaults/main.yml | 19 - .../roles/tap_interface/tasks/main.yml | 35 - metron-deployment/roles/yaf/defaults/main.yml | 29 - metron-deployment/roles/yaf/meta/main.yml | 22 - metron-deployment/roles/yaf/tasks/fixbuf.yml | 37 - metron-deployment/roles/yaf/tasks/main.yml | 20 - metron-deployment/roles/yaf/tasks/nic.yml | 19 - metron-deployment/roles/yaf/tasks/yaf.yml | 50 - .../roles/yaf/templates/start-yaf.sh | 25 - metron-deployment/roles/yaf/templates/yaf | 83 - metron-deployment/vagrant/README.md | 21 - .../vagrant/fastcapa-test-platform/.gitignore | 1 - .../vagrant/fastcapa-test-platform/README.md | 138 - .../centos-7.1/Vagrantfile | 72 - .../centos-7.1/ansible.cfg | 22 - .../centos-7.1/vars/main.yml | 52 - .../centos-7.4/Vagrantfile | 72 - .../centos-7.4/ansible.cfg | 22 - .../centos-7.4/vars/main.yml | 52 - .../vagrant/fastcapa-test-platform/playbook.yml | 49 - .../tasks/validate-messages-received.yml | 38 - .../tasks/validate-packets-sent.yml | 41 - .../vagrant/full-dev-platform/.gitignore | 1 - .../vagrant/full-dev-platform/README.md | 123 - .../vagrant/full-dev-platform/Vagrantfile | 92 - .../vagrant/full-dev-platform/ansible.cfg | 27 - .../full-dev-platform/run_ansible_role.sh | 20 - .../full-dev-platform/run_enrichment_role.sh | 20 - metron-interface/metron-rest/README.md | 6 +- 436 files changed, 61195 insertions(+), 60980 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 86d61e7..4858807 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ See the License for the specific language governing permissions and limitations under the License. --> [![Build Status](https://travis-ci.org/apache/metron.svg?branch=master)](https://travis-ci.org/apache/metron) - + # Apache Metron Metron integrates a variety of open source big data technologies in order @@ -27,36 +27,36 @@ the most current threat intelligence information to security telemetry within a single platform. For the latest information, please visit our website at http://metron.apache.org/ - + Metron can be divided into 4 areas: 1. **A mechanism to capture, store, and normalize any type of security telemetry at extremely high rates.** Because security telemetry is constantly -being generated, it requires a method for ingesting the data at high speeds +being generated, it requires a method for ingesting the data at high speeds and pushing it to various processing units for advanced computation and analytics. 2. **Real time processing and application of enrichments** such as threat intelligence, geolocation, and DNS information to telemetry being collected. The immediate application of this information to incoming telemetry provides -the context and situational awareness, as well as the who and where +the context and situational awareness, as well as the who and where information critical for investigation 3. **Efficient information storage** based on how the information will be used: - Logs and telemetry are stored such that they can be efficiently mined and analyzed for concise security visibility - - The ability to extract and reconstruct full packets helps an analyst answer -questions such as who the true attacker was, what data was leaked, and where + - The ability to extract and reconstruct full packets helps an analyst answer +questions such as who the true attacker was, what data was leaked, and where that data was sent - - Long-term storage not only increases visibility over time, but also enables -advanced analytics such as machine learning techniques to be used to create -models on the information. Incoming data can then be scored against these + - Long-term storage not only increases visibility over time, but also enables +advanced analytics such as machine learning techniques to be used to create +models on the information. Incoming data can then be scored against these stored models for advanced anomaly detection. -4. **An interface that gives a security investigator a centralized view of data -and alerts passed through the system.** Metron’s interface presents alert -summaries with threat intelligence and enrichment data specific to that alert -on one single page. Furthermore, advanced search capabilities and full packet -extraction tools are presented to the analyst for investigation without the +4. **An interface that gives a security investigator a centralized view of data +and alerts passed through the system.** Metron’s interface presents alert +summaries with threat intelligence and enrichment data specific to that alert +on one single page. Furthermore, advanced search capabilities and full packet +extraction tools are presented to the analyst for investigation without the need to pivot into additional tools. Big data is a natural fit for powerful security analytics. The Metron @@ -83,7 +83,7 @@ Option 3 is more likely to have the latest code. # Getting Started -To start exploring the capabilities of Apache Metron [follow these instructions to launch Metron in a single-node VM](metron-deployment/vagrant/full-dev-platform) on your own hardware. +To start exploring the capabilities of Apache Metron [follow these instructions to launch Metron in a single-node VM](metron-deployment/development/centos6) on your own hardware. # Building Metron @@ -146,4 +146,4 @@ Some useful utilities that cross all of these parts of the architecture: In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, it is necessary to add an additional field to the templates and mapping for existing sensors. -Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](./metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x) \ No newline at end of file +Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](./metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x) http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/dev-utilities/release-utils/metron-rc-check ---------------------------------------------------------------------- diff --git a/dev-utilities/release-utils/metron-rc-check b/dev-utilities/release-utils/metron-rc-check index da5412e..142cd39 100755 --- a/dev-utilities/release-utils/metron-rc-check +++ b/dev-utilities/release-utils/metron-rc-check @@ -260,7 +260,7 @@ echo "" read -p " run vagrant full_dev? [yN] " -n 1 -r echo if [[ $REPLY =~ ^[Yy]$ ]]; then - cd "$WORK/apache-metron-$METRON_VERSION-$RC/metron-deployment/vagrant/full-dev-platform" || exit 1 + cd "$WORK/apache-metron-$METRON_VERSION-$RC/metron-deployment/development/centos6" || exit 1 if [[ ${DID_BUILD} -ne 1 ]]; then vagrant up else http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-contrib/metron-docker/README.md ---------------------------------------------------------------------- diff --git a/metron-contrib/metron-docker/README.md b/metron-contrib/metron-docker/README.md index c290732..35364d4 100644 --- a/metron-contrib/metron-docker/README.md +++ b/metron-contrib/metron-docker/README.md @@ -19,12 +19,12 @@ limitations under the License. Metron Docker is a [Docker Compose](https://docs.docker.com/compose/overview/) application that is intended only for development and integration testing of Metron. These images can quickly spin-up the underlying components on which Apache Metron runs. -None of the core Metron components are setup or launched automatically with these Docker images. You will need to manually setup and start the Metron components that you require. You should not expect to see telemetry being parsed, enriched, or indexed. If you are looking to try-out, experiment or demo Metron capabilities on a single node, then the [Vagrant-driven VM](../../metron-deployment/vagrant/full-dev-platform) is what you need. Use this instead of Vagrant when: - +None of the core Metron components are setup or launched automatically with these Docker images. You will need to manually setup and start the Metron components that you require. You should not expect to see telemetry being parsed, enriched, or indexed. If you are looking to try-out, experiment or demo Metron capabilities on a single node, then the [Vagrant-driven VM](../../metron-deployment/development/centos6) is what you need. Use this instead of Vagrant when: + - You want an environment that can be built and spun up quickly - You need to frequently rebuild and restart services - You only need to test, troubleshoot or develop against a subset of services - + Metron Docker includes these images that have been customized for Metron: - Kafka (with Zookeeper) @@ -42,7 +42,7 @@ Install [Docker for Mac](https://docs.docker.com/docker-for-mac/) or [Docker for - Docker version 1.12.0 - docker-machine version 0.8.0 - docker-compose version 1.8.0 - + Build Metron from the top level directory with: ``` $ cd $METRON_HOME @@ -52,7 +52,7 @@ $ mvn clean install -DskipTests You are welcome to use an existing Docker host but we prefer one with more resources. You can create one of those with this script: ``` $ export METRON_DOCKER_HOME=$METRON_HOME/metron-contrib/metron-docker -$ cd $METRON_DOCKER_HOME +$ cd $METRON_DOCKER_HOME $ ./scripts/create-docker-machine.sh ``` @@ -161,7 +161,7 @@ $ docker-compose exec kafkazk ./bin/produce-data.sh Usage: produce-data.sh data_path topic [message_delay_in_seconds] # Stream data in TestData.txt to the 'test' Kafka topic at a frequency of 5 seconds (default is 1 second) -$ docker-compose exec kafkazk ./bin/produce-data.sh /data/TestData.txt test 5 +$ docker-compose exec kafkazk ./bin/produce-data.sh /data/TestData.txt test 5 ``` The Kafka/Zookeeper image comes with sample Bro and Squid data: @@ -269,4 +269,4 @@ $ curl -XGET http://192.168.99.100:9200/_cat/indices?v health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open .kibana 1 1 1 0 3.1kb 3.1kb yellow open bro_index_2016.12.19.18 5 1 180 0 475kb 475kb -``` \ No newline at end of file +``` http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/Kerberos-manual-setup.md ---------------------------------------------------------------------- diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md index 5d17cf0..47a63d8 100644 --- a/metron-deployment/Kerberos-manual-setup.md +++ b/metron-deployment/Kerberos-manual-setup.md @@ -34,7 +34,7 @@ This document provides instructions for kerberizing Metron's Vagrant-based devel Setup ----- -1. Deploy the [development environment.](vagrant/full-dev-platform/README.md). +1. Deploy the [development environment.](development/centos6/README.md). 1. Export the following environment variables. These need to be set for the remainder of the instructions. Replace `node1` with the appropriate hosts, if you are running Metron anywhere other than Vagrant. @@ -90,7 +90,7 @@ Setup a KDC cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts ``` -1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on [Full Dev](vagrant/full-dev-platform/README.md). +1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on a [single VM](development/centos6/README.md). Edit `/var/kerberos/krb5kdc/kdc.conf` and ensure the following is added to the `realm` section http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/README.md b/metron-deployment/README.md index 86d04e7..805aaf0 100644 --- a/metron-deployment/README.md +++ b/metron-deployment/README.md @@ -67,7 +67,7 @@ Running Metron within the resource constraints of a single VM is incredibly chal #### How? -To deploy Metron in a VM running on your computer, follow the instructions at [vagrant/full-dev-platform](vagrant/full-dev-platform) +To deploy Metron in a VM running on your computer, follow the instructions at [development/centos6](development/centos6). How do I build RPM packages? http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/amazon-ec2/ansible.cfg ---------------------------------------------------------------------- diff --git a/metron-deployment/amazon-ec2/ansible.cfg b/metron-deployment/amazon-ec2/ansible.cfg index 9f3a2fe..1f4f0ea 100644 --- a/metron-deployment/amazon-ec2/ansible.cfg +++ b/metron-deployment/amazon-ec2/ansible.cfg @@ -16,8 +16,8 @@ # [defaults] host_key_checking = False -library = ../extra_modules -roles_path = ../roles +library = ../ansible/extra_modules +roles_path = ../ansible/roles pipelining = True remote_user = centos forks = 20 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/amazon-ec2/conf/defaults.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/amazon-ec2/conf/defaults.yml b/metron-deployment/amazon-ec2/conf/defaults.yml index 124d7ab..b4e24f8 100644 --- a/metron-deployment/amazon-ec2/conf/defaults.yml +++ b/metron-deployment/amazon-ec2/conf/defaults.yml @@ -15,14 +15,6 @@ # limitations under the License. # --- -# which services should be started? -services_to_start: - - yaf - - snort - - snort-logs - - bro - - pcap-replay - # the ami for cent6 by region amis_by_region: us-east-1: ami-1c221e76 # US East (N. Virginia) http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/amazon-ec2/playbook.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/amazon-ec2/playbook.yml b/metron-deployment/amazon-ec2/playbook.yml index de51534..de64490 100644 --- a/metron-deployment/amazon-ec2/playbook.yml +++ b/metron-deployment/amazon-ec2/playbook.yml @@ -71,4 +71,4 @@ # # build the metron cluster # -- include: ../playbooks/metron_full_install.yml +- include: ../ansible/playbooks/metron_full_install.yml http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/README.md b/metron-deployment/ansible/README.md new file mode 100644 index 0000000..d35e755 --- /dev/null +++ b/metron-deployment/ansible/README.md @@ -0,0 +1,24 @@ + +# Ansible Assets + +This directory contains all of the shared Ansible assets used to deploy Metron in a [development environment](../development/README.md). The scope of what Ansible is used for is intentionally limited. The recommended means of deploying Metron is to use the [Metron MPack](../packaging/ambari/metron-mpack/README.md) for [Apache Ambari](https://ambari.apache.org/). + +Ansible is only used primarily to prepare the development environment for Ambari and for deploying a suite of test sensors to drive telemetry through Metron. The Ansible assets contained here are of limited use outside of this scenario. + +**Warning** It is not recommended that you use these assets for deploying Metron in your environment for either production or testing. Support for this use case cannot be provided. http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/extra_modules/ambari_cluster_state.py ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/extra_modules/ambari_cluster_state.py b/metron-deployment/ansible/extra_modules/ambari_cluster_state.py new file mode 100644 index 0000000..0542526 --- /dev/null +++ b/metron-deployment/ansible/extra_modules/ambari_cluster_state.py @@ -0,0 +1,395 @@ +#!/usr/bin/python +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +DOCUMENTATION = ''' +--- +module: ambari_cluster_state +version_added: "2.1" +author: Mark Bittmann (https://github.com/mbittmann) +short_description: Create, delete, start or stop an ambari cluster +description: + - Create, delete, start or stop an ambari cluster +options: + host: + description: + The hostname for the ambari web server + port: + description: + The port for the ambari web server + username: + description: + The username for the ambari web server + password: + description: + The name of the cluster in web server + required: yes + cluster_name: + description: + The name of the cluster in ambari + required: yes + cluster_state: + description: + The desired state for the ambari cluster ['present', 'absent', 'started', 'stopped']. Setting the cluster + state to absent will first stop the cluster. + required: yes + blueprint_var: + description: + The path to the file defining the cluster blueprint and host mapping. Required when state == 'present' + required: no + blueprint_name: + description: + The name of the blueprint. Required when state == 'present' + required: no + wait_for_complete: + description: + Whether to wait for the request to complete before returning. Default is False. + required: no + requirements: [ 'requests'] +''' + +EXAMPLES = ''' +# must use full relative path to any files in stored in roles/role_name/files/ +- name: Create a new ambari cluster + ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: present + blueprint_var: roles/my_role/files/blueprint.yml + blueprint_name: hadoop + wait_for_complete: True +- name: Start the ambari cluster + ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: started + wait_for_complete: True +- name: Stop the ambari cluster + ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: stopped + wait_for_complete: True +- name: Delete the ambari cluster + ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: absent +''' + +RETURN = ''' +results: + description: The content of the requests object returned from the RESTful call + returned: success + type: string +created_blueprint: + description: Whether a blueprint was created + returned: success + type: boolean +status: + description: The status of the blueprint creation process + returned: success + type: string +''' + +__author__ = 'mbittmann' + +import json +try: + import requests +except ImportError: + REQUESTS_FOUND = False +else: + REQUESTS_FOUND = True + + +def main(): + + argument_spec = dict( + host=dict(type='str', default=None, required=True), + port=dict(type='int', default=None, required=True), + username=dict(type='str', default=None, required=True), + password=dict(type='str', default=None, required=True), + cluster_name=dict(type='str', default=None, required=True), + cluster_state=dict(type='str', default=None, required=True, + choices=['present', 'absent', 'started', 'stopped']), + blueprint_var=dict(type='dict', required=False), + blueprint_name=dict(type='str', default=None, required=False), + configurations=dict(type='list', default=None, required=False), + wait_for_complete=dict(default=False, required=False, type='bool'), + ) + + required_together = ['blueprint_var', 'blueprint_name'] + + module = AnsibleModule( + argument_spec=argument_spec, + required_together=required_together + ) + + if not REQUESTS_FOUND: + module.fail_json( + msg='requests library is required for this module') + + p = module.params + + host = p.get('host') + port = p.get('port') + username = p.get('password') + password = p.get('password') + cluster_name = p.get('cluster_name') + cluster_state = p.get('cluster_state') + blueprint_name = p.get('blueprint_name') + wait_for_complete = p.get('wait_for_complete') + + ambari_url = 'http://{0}:{1}'.format(host, port) + + try: + if cluster_state in ['started', 'stopped']: + if not cluster_exists(ambari_url, username, password, cluster_name): + module.fail_json(msg="Cluster name {0} does not exist".format(cluster_name)) + state = '' + if cluster_state == 'started': + state = 'STARTED' + elif cluster_state == 'stopped': + state = 'INSTALLED' + + request = set_cluster_state(ambari_url, username, password, cluster_name, state) + if wait_for_complete: + try: + request_id = json.loads(request.content)['Requests']['id'] + except ValueError: + module.exit_json(changed=True, results=request.content) + status = wait_for_request_complete(ambari_url, username, password, cluster_name, request_id, 2) + if status != 'COMPLETED': + module.fail_json(msg="Request failed with status {0}".format(status)) + module.exit_json(changed=True, results=request.content) + elif cluster_state == 'absent': + if not cluster_exists(ambari_url, username, password, cluster_name): + module.exit_json(changed=False, msg='Skipping. Cluster does not exist') + if not can_delete_cluster(ambari_url, username, password, cluster_name): + request = set_cluster_state(ambari_url, username, password, cluster_name, 'INSTALLED') + request_id = json.loads(request.content)['Requests']['id'] + status = wait_for_request_complete(ambari_url, username, password, cluster_name, request_id, 2) + if status != 'COMPLETED': + module.fail_json(msg="Request failed with status {0}".format(status)) + request = delete_cluster(ambari_url, username, password, cluster_name) + module.exit_json(changed=True, results=request.content) + elif cluster_state == 'present': + if not p.get('blueprint_var') or not blueprint_name: # have neither name nor file + module.fail_json(msg="Must provide blueprint_var and blueprint_name when cluster_state=='present'") + + blueprint_var = p.get('blueprint_var') + blueprint, host_map = blueprint_var_to_ambari_converter(blueprint_var) + created_blueprint = False + + if not blueprint_exists(ambari_url, username, password, blueprint_name): + create_blueprint(ambari_url, username, password, blueprint_name, blueprint) + created_blueprint = True + + if cluster_exists(ambari_url, username, password, cluster_name): + module.exit_json(changed=False, msg='Cluster {0} already exists'.format(cluster_name), + created_blueprint=created_blueprint) + + configurations = p.get('configurations') + request = create_cluster(ambari_url, username, password, cluster_name, blueprint_name, configurations, host_map) + request_id = json.loads(request.content)['Requests']['id'] + if wait_for_complete: + status = wait_for_request_complete(ambari_url, username, password, cluster_name, request_id, 2) + if status != 'COMPLETED': + module.fail_json(msg="Request failed with status {0}".format(status)) + request_status = get_request_status(ambari_url, username, password, cluster_name, request_id) + module.exit_json(changed=True, results=request.content, + created_blueprint=created_blueprint, status=request_status) + + except requests.ConnectionError, e: + module.fail_json(msg="Could not connect to Ambari client: " + str(e.message)) + except Exception, e: + module.fail_json(msg="Ambari client exception occurred: " + str(e.message)) + + +def get_clusters(ambari_url, user, password): + r = get(ambari_url, user, password, '/api/v1/clusters') + if r.status_code != 200: + msg = 'Could not get cluster list: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + clusters = json.loads(r.content) + return clusters['items'] + + +def cluster_exists(ambari_url, user, password, cluster_name): + clusters = get_clusters(ambari_url, user, password) + return cluster_name in [item['Clusters']['cluster_name'] for item in clusters] + + +def set_cluster_state(ambari_url, user, password, cluster_name, cluster_state): + path = '/api/v1/clusters/{0}/services'.format(cluster_name) + request = {"RequestInfo": {"context": "Setting cluster state"}, + "Body": {"ServiceInfo": {"state": "{0}".format(cluster_state)}}} + payload = json.dumps(request) + r = put(ambari_url, user, password, path, payload) + if r.status_code not in [202, 200]: + msg = 'Could not set cluster state: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def create_cluster(ambari_url, user, password, cluster_name, blueprint_name, configurations, hosts_json): + path = '/api/v1/clusters/{0}'.format(cluster_name) + data = json.dumps({'blueprint': blueprint_name, 'configurations': configurations, 'host_groups': hosts_json}) + f = open('cluster.log', 'w') + f.write(data) + f.close() + r = post(ambari_url, user, password, path, data) + if r.status_code != 202: + msg = 'Could not create cluster: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def get_request_status(ambari_url, user, password, cluster_name, request_id): + path = '/api/v1/clusters/{0}/requests/{1}'.format(cluster_name, request_id) + r = get(ambari_url, user, password, path) + if r.status_code != 200: + msg = 'Could not get cluster request status: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + service = json.loads(r.content) + return service['Requests']['request_status'] + + +def wait_for_request_complete(ambari_url, user, password, cluster_name, request_id, sleep_time): + while True: + status = get_request_status(ambari_url, user, password, cluster_name, request_id) + if status == 'COMPLETED': + return status + elif status in ['FAILED', 'TIMEDOUT', 'ABORTED', 'SKIPPED_FAILED']: + return status + else: + time.sleep(sleep_time) + + +def can_delete_cluster(ambari_url, user, password, cluster_name): + path = '/api/v1/clusters/{0}/services?ServiceInfo/state=STARTED'.format(cluster_name) + r = get(ambari_url, user, password, path) + items = json.loads(r.content)['items'] + return len(items) > 0 + + +def get_blueprints(ambari_url, user, password): + path = '/api/v1/blueprints' + r = get(ambari_url, user, password, path) + if r.status_code != 200: + msg = 'Could not get blueprint list: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + + services = json.loads(r.content) + return services['items'] + + +def create_blueprint(ambari_url, user, password, blueprint_name, blueprint_data): + data = json.dumps(blueprint_data) + f = open('blueprint.log', 'w') + f.write(data) + f.close() + path = "/api/v1/blueprints/" + blueprint_name + r = post(ambari_url, user, password, path, data) + if r.status_code != 201: + msg = 'Could not create blueprint: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def blueprint_exists(ambari_url, user, password, blueprint_name): + blueprints = get_blueprints(ambari_url, user, password) + return blueprint_name in [item['Blueprints']['blueprint_name'] for item in blueprints] + + +def delete_cluster(ambari_url, user, password, cluster_name): + path = '/api/v1/clusters/{0}'.format(cluster_name) + r = delete(ambari_url, user, password, path) + if r.status_code != 200: + msg = 'Could not delete cluster: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def get(ambari_url, user, password, path): + r = requests.get(ambari_url + path, auth=(user, password)) + return r + + +def put(ambari_url, user, password, path, data): + headers = {'X-Requested-By': 'ambari'} + r = requests.put(ambari_url + path, data=data, auth=(user, password), headers=headers) + return r + + +def post(ambari_url, user, password, path, data): + headers = {'X-Requested-By': 'ambari'} + r = requests.post(ambari_url + path, data=data, auth=(user, password), headers=headers) + return r + + +def delete(ambari_url, user, password, path): + headers = {'X-Requested-By': 'ambari'} + r = requests.delete(ambari_url + path, auth=(user, password), headers=headers) + return r + + +def blueprint_var_to_ambari_converter(blueprint_var): + groups = blueprint_var['groups'] + new_groups = [] + host_map = [] + for group in groups: + components = [] + for component in group['components']: + components.append({'name': component}) + group['components'] = components + hosts = group.pop('hosts') + new_groups.append(group) + this_host_map = dict() + this_host_map['name'] = group['name'] + this_host_list = [{'fqdn': host} for host in hosts] + this_host_map['hosts'] = this_host_list + host_map.append(this_host_map) + blueprint = dict() + blueprint['configurations'] = blueprint_var['required_configurations'] + blueprint['host_groups'] = new_groups + blueprint['Blueprints'] = {'stack_name': blueprint_var['stack_name'], 'stack_version': blueprint_var['stack_version']} + return blueprint, host_map + +from ansible.module_utils.basic import * +if __name__ == '__main__': + main() http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/extra_modules/ambari_service_state.py ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/extra_modules/ambari_service_state.py b/metron-deployment/ansible/extra_modules/ambari_service_state.py new file mode 100644 index 0000000..6d44e8d --- /dev/null +++ b/metron-deployment/ansible/extra_modules/ambari_service_state.py @@ -0,0 +1,352 @@ +#!/usr/bin/python +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +DOCUMENTATION = ''' +--- +module: ambari_service_state +version_added: "2.1" +author: Apache Metron (https://metron.apache.org) +short_description: Start/Stop/Change Service or Component State +description: + - Start/Stop/Change Service or Component State +options: + host: + description: + The hostname for the ambari web server + port: + description: + The port for the ambari web server + username: + description: + The username for the ambari web server + password: + description: + The name of the cluster in web server + required: yes + cluster_name: + description: + The name of the cluster in ambari + required: yes + service_name: + description: + The name of the service to alter + required: no + component_name: + description: + The name of the component to alter + required: no + component_host: + description: + The host running the targeted component. Required when component_name is used. + required: no + state: + description: + The desired service/component state. + wait_for_complete: + description: + Whether to wait for the request to complete before returning. Default is False. + required: no + requirements: [ 'requests'] +''' + +EXAMPLES = ''' +# must use full relative path to any files in stored in roles/role_name/files/ +- name: Create a new ambari cluster + ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: present + blueprint_var: roles/my_role/files/blueprint.yml + blueprint_name: hadoop + wait_for_complete: True +- name: Start the ambari cluster + ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: started + wait_for_complete: True +- name: Stop the ambari cluster + ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: stopped + wait_for_complete: True +- name: Delete the ambari cluster + ambari_cluster_state: + host: localhost + port: 8080 + username: admin + password: admin + cluster_name: my_cluster + cluster_state: absent +''' + +RETURN = ''' +results: + description: The content of the requests object returned from the RESTful call + returned: success + type: string +''' + +__author__ = 'apachemetron' + +import json + +try: + import requests +except ImportError: + REQUESTS_FOUND = False +else: + REQUESTS_FOUND = True + + +def main(): + + argument_spec = dict( + host=dict(type='str', default=None, required=True), + port=dict(type='int', default=None, required=True), + username=dict(type='str', default=None, required=True), + password=dict(type='str', default=None, required=True), + cluster_name=dict(type='str', default=None, required=True), + state=dict(type='str', default=None, required=True, + choices=['started', 'stopped', 'deleted']), + service_name=dict(type='str', required=False), + component_name=dict(type='str', default=None, required=False), + component_host=dict(type='str', default=None, required=False), + wait_for_complete=dict(default=False, required=False, type='bool'), + ) + + required_together = ['component_name', 'component_host'] + + module = AnsibleModule( + argument_spec=argument_spec, + required_together=required_together + ) + + if not REQUESTS_FOUND: + module.fail_json( + msg='requests library is required for this module') + + p = module.params + + host = p.get('host') + port = p.get('port') + username = p.get('username') + password = p.get('password') + cluster_name = p.get('cluster_name') + state = p.get('state') + service_name = p.get('service_name') + component_name = p.get('component_name') + component_host = p.get('component_host') + wait_for_complete = p.get('wait_for_complete') + component_mode = False + ambari_url = 'http://{0}:{1}'.format(host, port) + + if component_name: + component_mode = True + + try: + if not cluster_exists(ambari_url, username, password, cluster_name): + module.fail_json(msg="Cluster name {0} does not exist".format(cluster_name)) + + if state in ['started', 'stopped', 'installed']: + desired_state = '' + + if state == 'started': + desired_state = 'STARTED' + elif state in ['stopped','installed']: + desired_state = 'INSTALLED' + + if component_mode: + if desired_state == 'INSTALLED': + if(can_add_component(ambari_url, username, password, cluster_name, component_name, component_host)): + add_component_to_host(ambari_url, username, password, cluster_name, component_name, component_host) + request = set_component_state(ambari_url, username, password, cluster_name, component_name, component_host, desired_state) + else: + request = set_service_state(ambari_url,username,password,cluster_name,service_name, desired_state) + if wait_for_complete: + try: + request_id = json.loads(request.content)['Requests']['id'] + except ValueError: + module.exit_json(changed=True, results=request.content) + status = wait_for_request_complete(ambari_url, username, password, cluster_name, request_id, 2) + if status != 'COMPLETED': + module.fail_json(msg="Request failed with status {0}".format(status)) + module.exit_json(changed=True, results=request.content) + + elif state == 'deleted': + if component_mode: + request = delete_component(ambari_url, username, password, cluster_name, component_name, component_host) + else: + request = delete_service(ambari_url,username,password,cluster_name,service_name) + module.exit_json(changed=True, results=request.content) + + except requests.ConnectionError, e: + module.fail_json(msg="Could not connect to Ambari client: " + str(e.message)) + except Exception, e: + module.fail_json(msg="Ambari client exception occurred: " + str(e.message)) + + +def get_clusters(ambari_url, user, password): + r = get(ambari_url, user, password, '/api/v1/clusters') + if r.status_code != 200: + msg = 'Could not get cluster list: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + clusters = json.loads(r.content) + return clusters['items'] + + +def cluster_exists(ambari_url, user, password, cluster_name): + clusters = get_clusters(ambari_url, user, password) + return cluster_name in [item['Clusters']['cluster_name'] for item in clusters] + + +def get_request_status(ambari_url, user, password, cluster_name, request_id): + path = '/api/v1/clusters/{0}/requests/{1}'.format(cluster_name, request_id) + r = get(ambari_url, user, password, path) + if r.status_code != 200: + msg = 'Could not get cluster request status: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + service = json.loads(r.content) + return service['Requests']['request_status'] + + +def wait_for_request_complete(ambari_url, user, password, cluster_name, request_id, sleep_time): + while True: + status = get_request_status(ambari_url, user, password, cluster_name, request_id) + if status == 'COMPLETED': + return status + elif status in ['FAILED', 'TIMEDOUT', 'ABORTED', 'SKIPPED_FAILED']: + return status + else: + time.sleep(sleep_time) + + +def set_service_state(ambari_url, user, password, cluster_name, service_name, desired_state): + path = '/api/v1/clusters/{0}/services/{1}'.format(cluster_name,service_name) + request = {"RequestInfo": {"context": "Setting {0} to {1} via REST".format(service_name,desired_state)}, + "Body": {"ServiceInfo": {"state": "{0}".format(desired_state)}}} + payload = json.dumps(request) + r = put(ambari_url, user, password, path, payload) + if r.status_code not in [202, 200]: + msg = 'Could not set service state: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def set_component_state(ambari_url, user, password, cluster_name, component_name, component_host, desired_state): + path = '/api/v1/clusters/{0}/hosts/{1}/host_components/{2}'.format(cluster_name,component_host,component_name) + request = {"RequestInfo": {"context": "Setting {0} to {1} via REST".format(component_name,desired_state)}, + "Body": {"HostRoles": {"state": "{0}".format(desired_state)}}} + payload = json.dumps(request) + r = put(ambari_url, user, password, path, payload) + if r.status_code not in [202, 200]: + msg = 'Could not set component state: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def delete_component(ambari_url, user, password, cluster_name, component_name, component_host): + enable_maint_mode(ambari_url, user, password, cluster_name, component_name, component_host) + path = '/api/v1/clusters/{0}/hosts/{1}/host_components/{2}'.format(cluster_name,component_host,component_name) + r = delete(ambari_url,user,password,path) + if r.status_code not in [202, 200]: + msg = 'Could not set service state: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def enable_maint_mode(ambari_url, user, password, cluster_name, component_name, component_host): + path = '/api/v1/clusters/{0}/hosts/{1}/host_components/{2}'.format(cluster_name,component_host,component_name) + request = {"RequestInfo":{"context":"Turn On Maintenance Mode for {0}".format(component_name)}, + "Body":{"HostRoles":{"maintenance_state":"ON"}}} + payload = json.dumps(request) + r = put(ambari_url, user, password, path, payload) + if r.status_code not in [202, 200]: + msg = 'Could not set maintenance mode: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def delete_service(ambari_url, user, password, cluster_name, service_name): + path = '/api/v1/clusters/{0}/services/{1}'.format(cluster_name,service_name) + r = delete(ambari_url,user,password,path) + if r.status_code not in [202, 200]: + msg = 'Could not delete service: request code {0}, \ + request message {1}'.format(r.status_code, r.content) + raise Exception(msg) + return r + + +def add_component_to_host(ambari_url, user, password, cluster_name, component_name, component_host): + path = '/api/v1/clusters/{0}/hosts/{1}/host_components/{2}'.format(cluster_name,component_host,component_name) + r = post(ambari_url, user, password, path,'') + if r.status_code not in [202,201,200]: + msg = 'Could not add {0} to host {1}: request code {2}, \ + request message {3}'.format(component_name,component_host,r.status_code, r.content) + raise Exception(msg) + return r + + +def can_add_component(ambari_url, user, password, cluster_name, component_name, component_host): + path = '/api/v1/clusters/{0}/hosts/{1}/host_components/{2}'.format(cluster_name,component_host,component_name) + r = get(ambari_url, user, password, path) + return r.status_code == 404 + + +def get(ambari_url, user, password, path): + r = requests.get(ambari_url + path, auth=(user, password)) + return r + + +def put(ambari_url, user, password, path, data): + headers = {'X-Requested-By': 'ambari'} + r = requests.put(ambari_url + path, data=data, auth=(user, password), headers=headers) + return r + + +def post(ambari_url, user, password, path, data): + headers = {'X-Requested-By': 'ambari'} + r = requests.post(ambari_url + path, data=data, auth=(user, password), headers=headers) + return r + + +def delete(ambari_url, user, password, path): + headers = {'X-Requested-By': 'ambari'} + r = requests.delete(ambari_url + path, auth=(user, password), headers=headers) + return r + + +from ansible.module_utils.basic import * +if __name__ == '__main__': + main() http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/playbooks/ambari_install.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/playbooks/ambari_install.yml b/metron-deployment/ansible/playbooks/ambari_install.yml new file mode 100644 index 0000000..5b802b3 --- /dev/null +++ b/metron-deployment/ansible/playbooks/ambari_install.yml @@ -0,0 +1,54 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +- hosts: ec2 + become: true + tasks: + - include_vars: ../../amazon-ec2/conf/defaults.yml + tags: + - ec2 + +- hosts: packer + become: true + tasks: + - include_vars: ../../development/centos6/ansible/inventory/group_vars/all + tags: + - packer + +- hosts: ambari_* + become: true + roles: + - role: ambari_common + tags: + - ambari-prereqs + - hdp-install + +- hosts: ambari_master + become: true + roles: + - role: ambari_master + tags: + - ambari-server + - hdp-install + +- hosts: ambari_slave + become: true + roles: + - role: ambari_slave + tags: + - ambari-agent + - hdp-install http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/playbooks/docker_probe_install.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/playbooks/docker_probe_install.yml b/metron-deployment/ansible/playbooks/docker_probe_install.yml new file mode 100644 index 0000000..7be779a --- /dev/null +++ b/metron-deployment/ansible/playbooks/docker_probe_install.yml @@ -0,0 +1,62 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +# +# sensors +# +- hosts: localhost + tasks: + - name: add container to inventory + add_host: + name: amb-server + ansible_connection: docker + groups: sensors + changed_when: false + tags: add-host + +- hosts: sensors + vars: + metron_version: 0.4.3 + metron_directory: /usr/metron/{{ metron_version }} + bro_version: "2.5.2" + fixbuf_version: "1.7.1" + yaf_version: "2.8.0" + daq_version: "2.0.6-1" + pycapa_repo: "https://github.com/OpenSOC/pycapa.git" + pycapa_home: "/opt/pycapa" + snort_version: "2.9.8.0-1" + snort_alert_csv_path: "/var/log/snort/alert.csv" + threat_intel_bulk_load: False + sensor_test_mode: True + install_pycapa: False + install_bro: True + install_snort: True + install_yaf: True + install_pcap_replay: True + sniff_interface: eth0 + pcap_replay_interface: "{{ sniff_interface }}" + pcapservice_port: 8081 + kafka_broker_url: amb4.service.consul:6667 + connection: docker + roles: + - { role: bro, tags: ['bro'] } + - { role: snort, tags: ['snort'] } + - { role: yaf, tags: ['yaf'] } + - { role: pcap_replay, tags: ['pcap-replay'] } + - { role: sensor-test-mode, tags: ['sensor-test-mode'] } + tags: + - sensors http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/playbooks/metron_build.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/playbooks/metron_build.yml b/metron-deployment/ansible/playbooks/metron_build.yml new file mode 100644 index 0000000..83eb352 --- /dev/null +++ b/metron-deployment/ansible/playbooks/metron_build.yml @@ -0,0 +1,23 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +- hosts: all + become: false + roles: + - role: metron-builder + tags: + - build http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/playbooks/metron_full_install.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/playbooks/metron_full_install.yml b/metron-deployment/ansible/playbooks/metron_full_install.yml new file mode 100644 index 0000000..b517671 --- /dev/null +++ b/metron-deployment/ansible/playbooks/metron_full_install.yml @@ -0,0 +1,46 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +- hosts: all + pre_tasks: + - name: Verify Ansible Version + fail: msg="Metron Requires Ansible 2.0.0.2 or 2.2.2.0, current version is {{ ansible_version }}" + when: "ansible_version.full | version_compare('2.2.2.0', '!=') and ansible_version.full | version_compare('2.0.0.2', '!=')" + +- include: metron_build.yml + tags: + - build + +- include: ambari_install.yml + tags: + - ambari + +- include: metron_install.yml + tags: + - metron + +- include: sensor_install.yml + +# +# deployment report +# +- hosts: metron + become: false + roles: + - { role: deployment-report } + tags: + - report http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/playbooks/metron_install.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/playbooks/metron_install.yml b/metron-deployment/ansible/playbooks/metron_install.yml new file mode 100644 index 0000000..49db5e6 --- /dev/null +++ b/metron-deployment/ansible/playbooks/metron_install.yml @@ -0,0 +1,75 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +- hosts: ec2 + become: true + tasks: + - include_vars: ../../amazon-ec2/conf/defaults.yml + tags: + - ec2 + +- hosts: packer + become: true + tasks: + - include_vars: ../../development/centos6/ansible/inventory/group_vars/all + tags: + - packer + +# +# all ambari slaves need the local repo setup, but most don't need the metron packages +# +- hosts: ambari_slave + become: true + roles: + - { role: metron-packages, packages_required: false } + tags: + - packages + +# +# push the metron packages (RPM or DEB) to the metron host(s) only +# +- hosts: metron + become: true + roles: + - { role: metron-packages, packages_required: true } + tags: + - packages + +# +# start installation of components in Ambari +# +- hosts: ambari_master + become: true + roles: + - role: ambari_config + tags: + - hdp-install + - hdp-deploy + +- hosts: ambari_master + become: true + roles: + - role: load_web_templates + tags: + - load_templates + +- hosts: pcap_server + become: true + roles: + - role: metron_pcapservice + tags: + - pcap-service http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/playbooks/sensor_install.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/playbooks/sensor_install.yml b/metron-deployment/ansible/playbooks/sensor_install.yml new file mode 100644 index 0000000..a683ccb --- /dev/null +++ b/metron-deployment/ansible/playbooks/sensor_install.yml @@ -0,0 +1,59 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +- hosts: ec2 + become: true + tasks: + - include_vars: ../../amazon-ec2/conf/defaults.yml + tags: + - ec2 + +- hosts: packer + become: true + tasks: + - include_vars: ../../development/centos6/ansible/inventory/group_vars/all + tags: + - packer + +# +# sensors +# +- hosts: sensors + become: true + roles: + - { role: ambari_gather_facts, tags: ['always'] } + - { role: tap_interface, tags: ['tap'] } + - { role: pycapa, tags: ['pycapa'] } + - { role: bro, tags: ['bro'] } + - { role: snort, tags: ['snort'] } + - { role: yaf, tags: ['yaf'] } + - { role: pcap_replay, tags: ['pcap-replay'] } + - { role: sensor-test-mode, tags: ['sensor-test-mode'] } + tags: + - sensors + +# +# sensor stubs - rather than deploying the sensors, deploy ligher weight +# stubs that mimic the behavior of the real sensors +# +- hosts: sensors + become: true + roles: + - { role: ambari_gather_facts } + - { role: sensor-stubs } + tags: + - sensor-stubs http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/roles/README.md ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/roles/README.md b/metron-deployment/ansible/roles/README.md new file mode 100644 index 0000000..1cb5878 --- /dev/null +++ b/metron-deployment/ansible/roles/README.md @@ -0,0 +1,24 @@ + +# Ansible Roles + +- Monit +- OpenTaxii +- Pcap Replay +- Sensor Stubs +- Sensor Test Mode http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/roles/ambari_common/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/roles/ambari_common/defaults/main.yml b/metron-deployment/ansible/roles/ambari_common/defaults/main.yml new file mode 100644 index 0000000..c04400e --- /dev/null +++ b/metron-deployment/ansible/roles/ambari_common/defaults/main.yml @@ -0,0 +1,23 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +hadoop_logrotate_frequency: daily +hadoop_logrotate_retention: 30 +centos_ambari_install_url: http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.4.2.0/ambari.repo +ubuntu_ambari_repo: http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.4.2.0 +ubuntu_elasticsearch_packages_repo: https://artifacts.elastic.co/packages/5.x/apt +ubuntu_elasticsearch_curator_repo: https://packages.elastic.co/curator/5/debian http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/ansible/roles/ambari_common/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/ansible/roles/ambari_common/meta/main.yml b/metron-deployment/ansible/roles/ambari_common/meta/main.yml new file mode 100644 index 0000000..d7e46d6 --- /dev/null +++ b/metron-deployment/ansible/roles/ambari_common/meta/main.yml @@ -0,0 +1,21 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +dependencies: + - libselinux-python + - epel + - ntp