Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 38C68200C36 for ; Fri, 10 Mar 2017 15:32:54 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 374BB160B79; Fri, 10 Mar 2017 14:32:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1BC71160B69 for ; Fri, 10 Mar 2017 15:32:51 +0100 (CET) Received: (qmail 78073 invoked by uid 500); 10 Mar 2017 14:32:51 -0000 Mailing-List: contact commits-help@metron.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.incubator.apache.org Delivered-To: mailing list commits@metron.incubator.apache.org Received: (qmail 78064 invoked by uid 99); 10 Mar 2017 14:32:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Mar 2017 14:32:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id AD244C0912 for ; Fri, 10 Mar 2017 14:32:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.069 X-Spam-Level: X-Spam-Status: No, score=-3.069 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id MzxVDCu-7gUH for ; Fri, 10 Mar 2017 14:32:39 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 570FF5F64B for ; Fri, 10 Mar 2017 14:32:37 +0000 (UTC) Received: (qmail 77453 invoked by uid 99); 10 Mar 2017 14:32:36 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Mar 2017 14:32:36 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 7DE46DFF30; Fri, 10 Mar 2017 14:32:36 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: leet@apache.org To: commits@metron.incubator.apache.org Date: Fri, 10 Mar 2017 14:32:37 -0000 Message-Id: In-Reply-To: <1c63d4f1e3c14b63aa8a3c35f32aa272@git.apache.org> References: <1c63d4f1e3c14b63aa8a3c35f32aa272@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [2/6] incubator-metron git commit: METRON-671 Refactor existing Ansible deployment to use Ambari MPack (dlyle via justinleet) closes apache/incubator-metron#436 archived-at: Fri, 10 Mar 2017 14:32:54 -0000 http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template b/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template deleted file mode 100644 index 8045c95..0000000 --- a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/bro_index.template +++ /dev/null @@ -1,218 +0,0 @@ -{ - "template": "bro_index*", - "mappings": { - "bro_doc": { - "_timestamp": { - "enabled": true - }, - "dynamic_templates": [ - { - "geo_location_point": { - "match": "enrichments:geo:*:location_point", - "match_mapping_type": "*", - "mapping": { - "type": "geo_point" - } - } - }, - { - "geo_country": { - "match": "enrichments:geo:*:country", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_city": { - "match": "enrichments:geo:*:city", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_location_id": { - "match": "enrichments:geo:*:locID", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_dma_code": { - "match": "enrichments:geo:*:dmaCode", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_postal_code": { - "match": "enrichments:geo:*:postalCode", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_latitude": { - "match": "enrichments:geo:*:latitude", - "match_mapping_type": "*", - "mapping": { - "type": "float" - } - } - }, - { - "geo_longitude": { - "match": "enrichments:geo:*:longitude", - "match_mapping_type": "*", - "mapping": { - "type": "float" - } - } - }, - { - "timestamps": { - "match": "*:ts", - "match_mapping_type": "*", - "mapping": { - "type": "date", - "format": "epoch_millis" - } - } - } - ], - "properties": { - "timestamp": { - "type": "date", - "format": "epoch_millis" - }, - "source:type": { - "type": "string", - "index": "not_analyzed" - }, - "ip_dst_addr": { - "type": "ip" - }, - "ip_dst_port": { - "type": "integer" - }, - "ip_src_addr": { - "type": "ip" - }, - "ip_src_port": { - "type": "integer" - }, - "status_code": { - "type": "integer" - }, - "method": { - "type": "string", - "index": "not_analyzed" - }, - "protocol": { - "type": "string", - "index": "not_analyzed" - }, - "request_body_len": { - "type": "long" - }, - "uri": { - "type": "string", - "index": "not_analyzed", - "ignore_above": 8191 - }, - "uid": { - "type": "string", - "index": "not_analyzed" - }, - "referrer": { - "type": "string", - "index": "not_analyzed" - }, - "trans_depth": { - "type": "integer" - }, - "host": { - "type": "string", - "index": "not_analyzed" - }, - "status_msg": { - "type": "string", - "index": "not_analyzed" - }, - "response_body_len": { - "type": "long" - }, - "user_agent": { - "type": "string" - }, - "query": { - "type": "string", - "index": "not_analyzed" - }, - "answers": { - "type": "string" - }, - "AA": { - "type": "boolean" - }, - "TC": { - "type": "boolean" - }, - "RA": { - "type": "boolean" - }, - "RD": { - "type": "boolean" - }, - "rejected": { - "type": "boolean" - }, - "qclass_name": { - "type": "string", - "index": "not_analyzed" - }, - "proto": { - "type": "string", - "index": "not_analyzed" - }, - "rcode": { - "type": "integer" - }, - "rcode_name": { - "type": "string", - "index": "not_analyzed" - }, - "trans_id": { - "type": "integer" - }, - "Z": { - "type": "integer" - }, - "qclass": { - "type": "integer" - }, - "qtype": { - "type": "integer" - }, - "qtype_name": { - "type": "string", - "index": "not_analyzed" - } - } - } - } -} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/snort_index.template ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/snort_index.template b/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/snort_index.template deleted file mode 100644 index bf943df..0000000 --- a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/snort_index.template +++ /dev/null @@ -1,183 +0,0 @@ -{ - "template": "snort_index*", - "mappings": { - "snort_doc": { - "_timestamp": { - "enabled": true - }, - "dynamic_templates": [ - { - "geo_location_point": { - "match": "enrichments:geo:*:location_point", - "match_mapping_type": "*", - "mapping": { - "type": "geo_point" - } - } - }, - { - "geo_country": { - "match": "enrichments:geo:*:country", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_city": { - "match": "enrichments:geo:*:city", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_location_id": { - "match": "enrichments:geo:*:locID", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_dma_code": { - "match": "enrichments:geo:*:dmaCode", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_postal_code": { - "match": "enrichments:geo:*:postalCode", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_latitude": { - "match": "enrichments:geo:*:latitude", - "match_mapping_type": "*", - "mapping": { - "type": "float" - } - } - }, - { - "geo_longitude": { - "match": "enrichments:geo:*:longitude", - "match_mapping_type": "*", - "mapping": { - "type": "float" - } - } - }, - { - "timestamps": { - "match": "*:ts", - "match_mapping_type": "*", - "mapping": { - "type": "date", - "format": "epoch_millis" - } - } - } - ], - "properties": { - "timestamp": { - "type": "date", - "format": "epoch_millis" - }, - "source:type": { - "type": "string", - "index": "not_analyzed" - }, - "ip_dst_addr": { - "type": "ip" - }, - "ip_dst_port": { - "type": "integer" - }, - "ip_src_addr": { - "type": "ip" - }, - "ip_src_port": { - "type": "integer" - }, - "dgmlen": { - "type": "integer" - }, - "ethdst": { - "type": "string", - "index": "not_analyzed" - }, - "ethlen": { - "type": "string", - "index": "not_analyzed" - }, - "ethsrc": { - "type": "string", - "index": "not_analyzed" - }, - "id": { - "type": "integer" - }, - "iplen": { - "type": "integer" - }, - "is_alert": { - "type": "boolean" - }, - "msg": { - "type": "string" - }, - "protocol": { - "type": "string", - "index": "not_analyzed" - }, - "sig_generator": { - "type": "string", - "index": "not_analyzed" - }, - "sig_id": { - "type": "integer" - }, - "sig_rev": { - "type": "string" - }, - "tcpack": { - "type": "string" - }, - "tcpflags": { - "type": "string" - }, - "tcpseq": { - "type": "string" - }, - "tcpwindow": { - "type": "string" - }, - "threat:triage:level": { - "type": "double" - }, - "tos": { - "type": "integer" - }, - "ttl": { - "type": "integer" - } - } - } - } -} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template b/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template deleted file mode 100644 index 7743afc..0000000 --- a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/yaf_index.template +++ /dev/null @@ -1,205 +0,0 @@ -{ - "template": "yaf_index*", - "mappings": { - "yaf_doc": { - "_timestamp": { - "enabled": true - }, - "dynamic_templates": [ - { - "geo_location_point": { - "match": "enrichments:geo:*:location_point", - "match_mapping_type": "*", - "mapping": { - "type": "geo_point" - } - } - }, - { - "geo_country": { - "match": "enrichments:geo:*:country", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_city": { - "match": "enrichments:geo:*:city", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_location_id": { - "match": "enrichments:geo:*:locID", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_dma_code": { - "match": "enrichments:geo:*:dmaCode", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_postal_code": { - "match": "enrichments:geo:*:postalCode", - "match_mapping_type": "*", - "mapping": { - "type": "string", - "index": "not_analyzed" - } - } - }, - { - "geo_latitude": { - "match": "enrichments:geo:*:latitude", - "match_mapping_type": "*", - "mapping": { - "type": "float" - } - } - }, - { - "geo_longitude": { - "match": "enrichments:geo:*:longitude", - "match_mapping_type": "*", - "mapping": { - "type": "float" - } - } - }, - { - "timestamps": { - "match": "*:ts", - "match_mapping_type": "*", - "mapping": { - "type": "date", - "format": "epoch_millis" - } - } - } - ], - "properties": { - "timestamp": { - "type": "date", - "format": "epoch_millis" - }, - "source:type": { - "type": "string", - "index": "not_analyzed" - }, - "ip_dst_addr": { - "type": "ip" - }, - "ip_dst_port": { - "type": "integer" - }, - "ip_src_addr": { - "type": "ip" - }, - "ip_src_port": { - "type": "integer" - }, - "start_time": { - "type": "date", - "format": "epoch_millis" - }, - "end_time": { - "type": "date", - "format": "epoch_millis" - }, - "duration": { - "type": "double" - }, - "rtt": { - "type": "double" - }, - "proto": { - "type": "string", - "index": "not_analyzed" - }, - "sip": { - "type": "string", - "index": "not_analyzed" - }, - "sp": { - "type": "string", - "index": "not_analyzed" - }, - "dip": { - "type": "string", - "index": "not_analyzed" - }, - "dp": { - "type": "string", - "index": "not_analyzed" - }, - "iflags": { - "type": "string", - "index": "not_analyzed" - }, - "uflags": { - "type": "string", - "index": "not_analyzed" - }, - "riflags": { - "type": "string" - }, - "ruflags": { - "type": "string", - "index": "not_analyzed" - }, - "isn": { - "type": "string", - "index": "not_analyzed" - }, - "risn": { - "type": "string", - "index": "not_analyzed" - }, - "tag": { - "type": "string", - "index": "not_analyzed" - }, - "rtag": { - "type": "string", - "index": "not_analyzed" - }, - "pkt": { - "type": "integer" - }, - "oct": { - "type": "integer" - }, - "rpkt": { - "type": "integer" - }, - "roct": { - "type": "integer" - }, - "app": { - "type": "string", - "index": "not_analyzed" - }, - "end-reason": { - "type": "string" - } - } - } - } -} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_elasticsearch_templates/tasks/load_templates.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_elasticsearch_templates/tasks/load_templates.yml b/metron-deployment/roles/metron_elasticsearch_templates/tasks/load_templates.yml deleted file mode 100644 index 812569d..0000000 --- a/metron-deployment/roles/metron_elasticsearch_templates/tasks/load_templates.yml +++ /dev/null @@ -1,53 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Start Elasticsearch - service: name=elasticsearch state=started - -- name : Wait for Elasticsearch Host to Start - wait_for: - host: "{{ inventory_hostname }}" - port: "{{ elasticsearch_web_port }}" - delay: 10 - timeout: 300 - -- name: Wait for Index to Become Available - uri: - url: "http://{{ inventory_hostname }}:{{ elasticsearch_web_port }}/_cat/health" - method: GET - status_code: 200 - return_content: yes - register: result - until: result.content.find("green") != -1 or result.content.find("yellow") != -1 - retries: 10 - delay: 60 - -- name: Add Elasticsearch templates for topologies - uri: - url: "http://{{ inventory_hostname }}:{{ elasticsearch_web_port }}/_template/{{ item | basename | replace('.template','') }}" - method: PUT - body: "{{ lookup('file',item) }}" - status_code: 200 - with_fileglob: ./files/es_templates/*.template - -- name: Validate Elasticsearch templates - uri: - url: "http://{{ inventory_hostname }}:{{ elasticsearch_web_port }}/_template/{{ item | basename | replace('.template','') }}" - method: HEAD - body: "{{ lookup('file',item) }}" - status_code: 200 - with_fileglob: ./files/es_templates/*.template http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_elasticsearch_templates/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_elasticsearch_templates/tasks/main.yml b/metron-deployment/roles/metron_elasticsearch_templates/tasks/main.yml deleted file mode 100644 index 61fb625..0000000 --- a/metron-deployment/roles/metron_elasticsearch_templates/tasks/main.yml +++ /dev/null @@ -1,18 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- include: load_templates.yml http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_hbase_tables/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_hbase_tables/defaults/main.yml b/metron-deployment/roles/metron_hbase_tables/defaults/main.yml deleted file mode 100644 index 9d40d1b..0000000 --- a/metron-deployment/roles/metron_hbase_tables/defaults/main.yml +++ /dev/null @@ -1,22 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -hbase_tables_to_create: - - pcap - - access_tracker - - threatintel - - enrichment http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_hbase_tables/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_hbase_tables/meta/main.yml b/metron-deployment/roles/metron_hbase_tables/meta/main.yml deleted file mode 100644 index c3d807b..0000000 --- a/metron-deployment/roles/metron_hbase_tables/meta/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - ambari_gather_facts http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_hbase_tables/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_hbase_tables/tasks/main.yml b/metron-deployment/roles/metron_hbase_tables/tasks/main.yml deleted file mode 100644 index cc86c09..0000000 --- a/metron-deployment/roles/metron_hbase_tables/tasks/main.yml +++ /dev/null @@ -1,24 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -# must run on hadoop host -- name: Create the HBase tables required for Metron - shell: echo "create '{{ item }}','t'" | hbase shell -n - with_items: "{{ hbase_tables_to_create }}" - register: out - failed_when: out.rc != 0 and ("Table already exists" not in out.stdout) - changed_when: ("Table already exists" not in out.stdout) http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_kafka_topics/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_kafka_topics/defaults/main.yml b/metron-deployment/roles/metron_kafka_topics/defaults/main.yml deleted file mode 100644 index 4a97a8e..0000000 --- a/metron-deployment/roles/metron_kafka_topics/defaults/main.yml +++ /dev/null @@ -1,30 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -kafka_home: /usr/hdp/current/kafka-broker/ -topics_to_create: - - { topic: "pcap", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "bro", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "yaf", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "snort", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "enrichments", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "enrichments_error", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "threatintel_error", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "parser_invalid", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "parser_error", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "indexing", num_partitions: 1, replication_factor: 1, retention_gb: 10 } - - { topic: "indexing_error", num_partitions: 1, replication_factor: 1, retention_gb: 10 } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_kafka_topics/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_kafka_topics/meta/main.yml b/metron-deployment/roles/metron_kafka_topics/meta/main.yml deleted file mode 100644 index c3d807b..0000000 --- a/metron-deployment/roles/metron_kafka_topics/meta/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - ambari_gather_facts http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/metron_kafka_topics/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_kafka_topics/tasks/main.yml b/metron-deployment/roles/metron_kafka_topics/tasks/main.yml deleted file mode 100644 index 8d3cb0a..0000000 --- a/metron-deployment/roles/metron_kafka_topics/tasks/main.yml +++ /dev/null @@ -1,30 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Create the Kafka topics required for Metron - shell: > - {{ kafka_home }}/bin/kafka-topics.sh \ - --zookeeper {{ zookeeper_url }} \ - --create \ - --topic {{ item.topic }} \ - --partitions {{ item.num_partitions }} \ - --replication-factor {{ item.replication_factor }} \ - --config retention.bytes={{ item.retention_gb * 1024 * 1024 * 1024 }} - with_items: "{{ topics_to_create }}" - register: out - failed_when: out.rc != 0 and ("already exists" not in out.stderr) - changed_when: ("already exists" not in out.stderr) http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/defaults/main.yml b/metron-deployment/roles/monit/defaults/main.yml index c1c7583..651aa58 100644 --- a/metron-deployment/roles/monit/defaults/main.yml +++ b/metron-deployment/roles/monit/defaults/main.yml @@ -19,9 +19,6 @@ monit_home: /usr/local/monit monit_config_home: /etc/monit.d monit_user: admin monit_pass: monit -topology_start_timeout: 120 -topology_stop_timeout: 120 bro_pid_file: /usr/local/bro/spool/bro/.pid -elasticsearch_pid_file: /var/run/elasticsearch/elasticsearch.pid snort_alert_csv_path: /var/log/snort/alert.csv http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/tasks/main.yml b/metron-deployment/roles/monit/tasks/main.yml index 3718797..9c1d75d 100644 --- a/metron-deployment/roles/monit/tasks/main.yml +++ b/metron-deployment/roles/monit/tasks/main.yml @@ -17,7 +17,6 @@ # --- - include: monit.yml -- include: scripts.yml - include: monit-definitions.yml - include: monit-sensor-definitions.yml @@ -26,4 +25,4 @@ - include: monit-stub-definitions.yml tags: - - sensor-stubs \ No newline at end of file + - sensor-stubs http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/tasks/monit-definitions.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/tasks/monit-definitions.yml b/metron-deployment/roles/monit/tasks/monit-definitions.yml index b6ea821..c9f5f07 100644 --- a/metron-deployment/roles/monit/tasks/monit-definitions.yml +++ b/metron-deployment/roles/monit/tasks/monit-definitions.yml @@ -16,26 +16,6 @@ # limitations under the License. # --- -- name: Create monit definition for elasticsearch - template: src=monit/elasticsearch.monit dest={{ monit_config_home }}/elasticsearch.monit - when: ("search" in group_names) and (install_elasticsearch | default(True)) - -- name: Create monit definition for indexing with elasticsearch - template: src=monit/indexing-elasticsearch.monit dest={{ monit_config_home }}/indexing-elasticsearch.monit - when: ("enrichment" in group_names) and (install_elasticsearch | default(True)) - -- name: Create monit definition for enrichment - template: src=monit/enrichment.monit dest={{ monit_config_home }}/enrichment.monit - when: ("enrichment" in group_names) - -- name: Create monit definition for kibana - template: src=monit/kibana.monit dest={{ monit_config_home }}/kibana.monit - when: ("web" in group_names) and (install_elasticsearch | default(True)) - -- name: Create monit definition for parsers - template: src=monit/parsers.monit dest={{ monit_config_home }}/parsers.monit - when: ("enrichment" in group_names) - - name: Create monit definition for pcap-replay template: src=monit/pcap-replay.monit dest={{ monit_config_home }}/pcap-replay.monit when: ("sensors" in group_names) and (install_pcap_replay | default(False)) http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/tasks/scripts.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/tasks/scripts.yml b/metron-deployment/roles/monit/tasks/scripts.yml deleted file mode 100644 index 43901d2..0000000 --- a/metron-deployment/roles/monit/tasks/scripts.yml +++ /dev/null @@ -1,55 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Create monit home - file: path={{ monit_home }} state=directory mode=0755 - -- name: Deploy parser topology start scripts - template: - src: scripts/start_topology.sh - dest: "{{ monit_home }}/start_{{ item }}_topology.sh" - mode: 0755 - with_items: - - yaf - - snort - - bro - -- name: Deploy topology stop scripts - template: - src: scripts/stop_topology.sh - dest: "{{ monit_home }}/stop_{{ item }}_topology.sh" - mode: 0755 - with_items: - - yaf - - snort - - bro - - pcap - - enrichment - - indexing - -- name: Deploy topology status scripts - template: - src: scripts/status_topology.sh - dest: "{{ monit_home }}/status_{{ item }}_topology.sh" - mode: 0755 - with_items: - - yaf - - snort - - bro - - pcap - - enrichment - - indexing http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/monit/elasticsearch.monit ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/monit/elasticsearch.monit b/metron-deployment/roles/monit/templates/monit/elasticsearch.monit deleted file mode 100644 index 805c3cb..0000000 --- a/metron-deployment/roles/monit/templates/monit/elasticsearch.monit +++ /dev/null @@ -1,23 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -check process elasticsearch with pidfile {{ elasticsearch_pid_file }} - start program = "/etc/init.d/elasticsearch start" - stop program = "/etc/init.d/elasticsearch stop" - if does not exist then restart - group enrichments - group search - group metron http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/monit/enrichment.monit ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/monit/enrichment.monit b/metron-deployment/roles/monit/templates/monit/enrichment.monit deleted file mode 100644 index c1c61d1..0000000 --- a/metron-deployment/roles/monit/templates/monit/enrichment.monit +++ /dev/null @@ -1,26 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -check program enrichment with path "{{ monit_home }}/status_enrichment_topology.sh" - start program "{{ metron_directory }}/bin/start_enrichment_topology.sh" with timeout {{ topology_start_timeout }} seconds - stop program "{{ monit_home }}/stop_enrichment_topology.sh" with timeout {{ topology_stop_timeout }} seconds - if status != 0 then restart - group yaf - group bro - group snort - group enrichments - group metron http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/monit/indexing-elasticsearch.monit ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/monit/indexing-elasticsearch.monit b/metron-deployment/roles/monit/templates/monit/indexing-elasticsearch.monit deleted file mode 100644 index af60f44..0000000 --- a/metron-deployment/roles/monit/templates/monit/indexing-elasticsearch.monit +++ /dev/null @@ -1,26 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -check program indexing with path "{{ monit_home }}/status_indexing_topology.sh" - start program "{{ metron_directory }}/bin/start_elasticsearch_topology.sh" with timeout {{ topology_start_timeout }} seconds - stop program "{{ monit_home }}/stop_indexing_topology.sh" with timeout {{ topology_stop_timeout }} seconds - if status != 0 then restart - group yaf - group bro - group snort - group enrichments - group metron http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/monit/indexing-solr.monit ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/monit/indexing-solr.monit b/metron-deployment/roles/monit/templates/monit/indexing-solr.monit deleted file mode 100644 index 6c7a508..0000000 --- a/metron-deployment/roles/monit/templates/monit/indexing-solr.monit +++ /dev/null @@ -1,26 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -check program indexing with path "{{ monit_home }}/status_indexing_topology.sh" - start program "{{ metron_directory }}/bin/start_solr_topology.sh" with timeout {{ topology_start_timeout }} seconds - stop program "{{ monit_home }}/stop_indexing_topology.sh" with timeout {{ topology_stop_timeout }} seconds - if status != 0 then restart - group yaf - group bro - group snort - group enrichments - group metron http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/monit/kibana.monit ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/monit/kibana.monit b/metron-deployment/roles/monit/templates/monit/kibana.monit deleted file mode 100644 index 41b4cb9..0000000 --- a/metron-deployment/roles/monit/templates/monit/kibana.monit +++ /dev/null @@ -1,22 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -check process kibana matching "/opt/kibana/bin/../node/bin/node /opt/kibana/bin/../src/cli" - start program = "/etc/init.d/kibana start" - stop program = "/etc/init.d/kibana stop" - if does not exist then restart - group web - group metron http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/monit/parsers.monit ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/monit/parsers.monit b/metron-deployment/roles/monit/templates/monit/parsers.monit deleted file mode 100644 index 1eff8b6..0000000 --- a/metron-deployment/roles/monit/templates/monit/parsers.monit +++ /dev/null @@ -1,49 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -check program pcap-parser with path "{{ monit_home }}/status_pcap_topology.sh" - start program "{{ metron_directory }}/bin/start_pcap_topology.sh" with timeout {{ topology_start_timeout }} seconds - stop program "{{ monit_home }}/stop_pcap_topology.sh" with timeout {{ topology_stop_timeout }} seconds - if status != 0 then restart - group pcap - group parsers - group metron - -check program yaf-parser with path "{{ monit_home }}/status_yaf_topology.sh" - start program "{{ monit_home }}/start_yaf_topology.sh" with timeout {{ topology_start_timeout }} seconds - stop program "{{ monit_home }}/stop_yaf_topology.sh" with timeout {{ topology_stop_timeout }} seconds - if status != 0 then restart - group yaf - group parsers - group metron - -check program bro-parser with path "{{ monit_home }}/status_bro_topology.sh" - start program "{{ monit_home }}/start_bro_topology.sh" with timeout {{ topology_start_timeout }} seconds - stop program "{{ monit_home }}/stop_bro_topology.sh" with timeout {{ topology_stop_timeout }} seconds - if status != 0 then restart - group bro - group parsers - group metron - -check program snort-parser with path "{{ monit_home }}/status_snort_topology.sh" - start program "{{ monit_home }}/start_snort_topology.sh" with timeout {{ topology_start_timeout }} seconds - stop program "{{ monit_home }}/stop_snort_topology.sh" with timeout {{ topology_stop_timeout }} seconds - if status != 0 then restart - group snort - group parsers - group metron http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/scripts/start_enrichment_topology.sh ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/scripts/start_enrichment_topology.sh b/metron-deployment/roles/monit/templates/scripts/start_enrichment_topology.sh deleted file mode 100644 index e170460..0000000 --- a/metron-deployment/roles/monit/templates/scripts/start_enrichment_topology.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# -# start the enrichment topology -# -export METRON_VERSION={{ metron_version }} -export METRON_HOME={{ metron_directory }} -{{ item }} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/scripts/start_topology.sh ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/scripts/start_topology.sh b/metron-deployment/roles/monit/templates/scripts/start_topology.sh deleted file mode 100644 index 5e5286d..0000000 --- a/metron-deployment/roles/monit/templates/scripts/start_topology.sh +++ /dev/null @@ -1,25 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# -# start a storm topology -# -export METRON_VERSION={{ metron_version }} -export METRON_HOME={{ metron_directory }} -$METRON_HOME/bin/start_parser_topology.sh -k {{ kafka_broker_url }} -z {{ zookeeper_url }} -s {{ item }} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/scripts/status_topology.sh ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/scripts/status_topology.sh b/metron-deployment/roles/monit/templates/scripts/status_topology.sh deleted file mode 100644 index 67e9373..0000000 --- a/metron-deployment/roles/monit/templates/scripts/status_topology.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# -# is a given storm topology running? -# - -TOPOLOGY={{ item }} -STATUS=`storm list | grep $TOPOLOGY | awk '{print $2}'` - -if [ "ACTIVE" = "$STATUS" ]; then - echo "Running: $TOPOLOGY" - exit 0 -else - echo "Stopped: $TOPOLOGY" - exit 1 -fi http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/monit/templates/scripts/stop_topology.sh ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/monit/templates/scripts/stop_topology.sh b/metron-deployment/roles/monit/templates/scripts/stop_topology.sh deleted file mode 100644 index d864621..0000000 --- a/metron-deployment/roles/monit/templates/scripts/stop_topology.sh +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# -# stop a storm topology -# -storm kill {{ item }} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/quick_dev/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/quick_dev/meta/main.yml b/metron-deployment/roles/quick_dev/meta/main.yml new file mode 100644 index 0000000..c3d807b --- /dev/null +++ b/metron-deployment/roles/quick_dev/meta/main.yml @@ -0,0 +1,19 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +dependencies: + - ambari_gather_facts http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/quick_dev/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/quick_dev/tasks/main.yml b/metron-deployment/roles/quick_dev/tasks/main.yml new file mode 100644 index 0000000..0fa0a23 --- /dev/null +++ b/metron-deployment/roles/quick_dev/tasks/main.yml @@ -0,0 +1,70 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +--- +- name: Delete the Metron Components from Ambari + ambari_service_state: + host: "{{ groups.ambari_master[0] }}" + port: "{{ ambari_port }}" + username: "{{ ambari_user }}" + password: "{{ ambari_password }}" + cluster_name: "{{ cluster_name }}" + state: deleted + component_name: "{{ item }}" + component_host: "{{ inventory_hostname }}" + with_items: + - METRON_ENRICHMENT_MASTER + - METRON_INDEXING + - METRON_PARSERS + +- name: Remove the Metron packages + package: + name: "{{ item }}" + state: absent + with_items: + - metron-common + - metron-data-management + - metron-parsers + - metron-enrichment + - metron-indexing + - metron-elasticsearch + +- name: Re-install the Metron Packages via Ambari + ambari_service_state: + host: "{{ groups.ambari_master[0] }}" + port: "{{ ambari_port }}" + username: "{{ ambari_user }}" + password: "{{ ambari_password }}" + cluster_name: "{{ cluster_name }}" + state: stopped + component_name: "{{ item }}" + component_host: "{{ inventory_hostname }}" + wait_for_complete: True + with_items: + - METRON_ENRICHMENT_MASTER + - METRON_INDEXING + - METRON_PARSERS + +- name: Start the ambari cluster + ambari_cluster_state: + host: "{{ groups.ambari_master[0] }}" + port: "{{ ambari_port }}" + username: "{{ ambari_user }}" + password: "{{ ambari_password }}" + cluster_name: "{{ cluster_name }}" + cluster_state: started + wait_for_complete: True + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/solr/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/solr/defaults/main.yml b/metron-deployment/roles/solr/defaults/main.yml deleted file mode 100644 index b40d534..0000000 --- a/metron-deployment/roles/solr/defaults/main.yml +++ /dev/null @@ -1,29 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -rhel_hdp_utils_install_url: http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6/hdp-util.repo -solr_install_path: /opt/lucidworks-hdpsearch/solr -solr_user: solr -solr_collection_name: Metron -solr_config_dir: "{{ solr_install_path }}/server/solr/configsets/basic_configs/conf" -solr_bin_dir: "/opt/lucidworks-hdpsearch/solr/bin" -solr_config_name: "metron_conf" -solr_number_shards: "{{ groups['search'] | length }}" -solr_replication_factor: 1 -solr_autoSoftCommit_maxTime: 60 -solr_cmd: "{{ solr_bin_dir}}/solr create_collection -c {{ solr_collection_name }} -d {{ solr_config_dir }} -n {{ solr_config_name }} -shards {{ solr_number_shards }} -replicationFactor {{ solr_replication_factor }}" -hdp_utils_repo_path: /etc/yum.repos.d/HDP-UTILS.repo \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/solr/files/schema.xml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/solr/files/schema.xml b/metron-deployment/roles/solr/files/schema.xml deleted file mode 100644 index 43452a2..0000000 --- a/metron-deployment/roles/solr/files/schema.xml +++ /dev/null @@ -1,191 +0,0 @@ - - - - - - - - ; - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - id - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/solr/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/solr/meta/main.yml b/metron-deployment/roles/solr/meta/main.yml deleted file mode 100644 index 454dd37..0000000 --- a/metron-deployment/roles/solr/meta/main.yml +++ /dev/null @@ -1,21 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - ambari_gather_facts - - java_jdk - http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/solr/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/solr/tasks/main.yml b/metron-deployment/roles/solr/tasks/main.yml deleted file mode 100644 index cfbb6b5..0000000 --- a/metron-deployment/roles/solr/tasks/main.yml +++ /dev/null @@ -1,74 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Check for Metron jar path - stat: path={{ hdp_utils_repo_path }} - register: hdp_utils - - -- name: Install HDP-UTILs Repo - get_url: - url: "{{ rhel_hdp_utils_install_url }}" - dest: /etc/yum.repos.d/HDP-UTILS.repo - when: hdp_utils.stat.exists == False - -- name: Install HDP-UTIL gpg key - rpm_key: - state: present - key: http://pgp.mit.edu/pks/lookup?op=get&search=0xB9733A7A07513CAD - when: hdp_utils.stat.exists == False - -- name: Install Solr - yum: - name: lucidworks-hdpsearch - state: present - -- name: Create solr.xml from template - template: - src: solr.xml - dest: "{{ solr_install_path }}/server/solr" - mode: 0644 - owner: "{{ solr_user }}" - group: "{{ solr_user }}" - -- name: Copy solrschema.xml to {{ inventory_hostname }} - copy: - src: schema.xml - dest: "{{ solr_config_dir }}" - mode: 0644 - owner: "{{ solr_user }}" - group: "{{ solr_user }}" - -- name: Create solrconfig.xml from template - template: - src: solrconfig.xml - dest: "{{ solr_config_dir }}" - mode: 0644 - owner: "{{ solr_user }}" - group: "{{ solr_user }}" - -- name: Start Solr - service: - name: solr - state: restarted - enabled: yes - -- name: Create Collection {{ solr_collection_name }} with {{ solr_number_shards }} shard(s) and replication factor {{ solr_replication_factor }} - shell: "{{ solr_cmd }}" - ignore_errors: yes - register: result - failed_when: result.rc == 1 and result.stderr.find("already exists!") == -1 http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/68a334a8/metron-deployment/roles/solr/templates/solr.xml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/solr/templates/solr.xml b/metron-deployment/roles/solr/templates/solr.xml deleted file mode 100644 index 407df13..0000000 --- a/metron-deployment/roles/solr/templates/solr.xml +++ /dev/null @@ -1,52 +0,0 @@ - - - - - - - - - - ${host:} - ${jetty.port:8983} - ${hostContext:solr} - - ${genericCoreNodeNames:true} - - {{ zookeeper_url }} - ${zkClientTimeout:30000} - ${distribUpdateSoTimeout:600000} - ${distribUpdateConnTimeout:60000} - - - - - ${socketTimeout:600000} - ${connTimeout:60000} - - -