From commits-return-2626-archive-asf-public=cust-asf.ponee.io@metron.apache.org Tue Jan 30 15:53:49 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id E3C5318067A for ; Tue, 30 Jan 2018 15:53:48 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D231F160C42; Tue, 30 Jan 2018 14:53:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 349FA160C68 for ; Tue, 30 Jan 2018 15:53:46 +0100 (CET) Received: (qmail 28535 invoked by uid 500); 30 Jan 2018 14:53:45 -0000 Mailing-List: contact commits-help@metron.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.apache.org Delivered-To: mailing list commits@metron.apache.org Received: (qmail 28411 invoked by uid 99); 30 Jan 2018 14:53:45 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jan 2018 14:53:45 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 8DF7EF4DBD; Tue, 30 Jan 2018 14:53:44 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: leet@apache.org To: commits@metron.apache.org Date: Tue, 30 Jan 2018 14:53:59 -0000 Message-Id: <4504042da0fd4874bedcab63db782d45@git.apache.org> In-Reply-To: <6fa45d72fbfe4b53a348c4ee942be0bd@git.apache.org> References: <6fa45d72fbfe4b53a348c4ee942be0bd@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [17/31] metron git commit: METRON-1370 Create Full Dev Equivalent for Ubuntu (nickwallen via cestella) closes apache/incubator-metron#903 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/bro/vars/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/bro/vars/main.yml b/metron-deployment/roles/bro/vars/main.yml deleted file mode 100644 index 11d6c31..0000000 --- a/metron-deployment/roles/bro/vars/main.yml +++ /dev/null @@ -1,34 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -bro_home: /usr/local/bro -bro_bin: "{{ bro_home }}/bin" -bro_version: 2.5.2 -bro_daemon_log: /var/log/bro.log -bro_topic: bro - -# Bro cronjob -bro_crontab_minutes: 0-59/5 -bro_crontab_job: "{{ bro_home }}/bin/broctl cron" -bro_clean_job: "rm -rf {{ bro_home }}/spool/tmp/*" - -# Bro kafka plugin -metron_bro_plugin_kafka_version: "0.1" -python27_bin: /opt/rh/python27/root/usr/bin -python27_lib: /opt/rh/python27/root/usr/lib64 -git29_bin: /opt/rh/rh-git29/root/usr/bin -httpd24_lib: /opt/rh/httpd24/root/usr/lib64 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/build-tools/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/build-tools/meta/main.yml b/metron-deployment/roles/build-tools/meta/main.yml deleted file mode 100644 index ddf6aa9..0000000 --- a/metron-deployment/roles/build-tools/meta/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - java_jdk http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/build-tools/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/build-tools/tasks/main.yml b/metron-deployment/roles/build-tools/tasks/main.yml deleted file mode 100644 index c47ef43..0000000 --- a/metron-deployment/roles/build-tools/tasks/main.yml +++ /dev/null @@ -1,34 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Install Build Tools - yum: name={{ item }} - with_items: - - "@Development tools" - - libdnet-devel - - rpm-build - - libpcap - - libpcap-devel - - pcre - - pcre-devel - - zlib - - zlib-devel - - glib2-devel - register: result - until: result.rc == 0 - retries: 5 - delay: 10 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/deployment-report/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/deployment-report/defaults/main.yml b/metron-deployment/roles/deployment-report/defaults/main.yml deleted file mode 100644 index 0a8afb8..0000000 --- a/metron-deployment/roles/deployment-report/defaults/main.yml +++ /dev/null @@ -1,18 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -timeout_secs: 120 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/deployment-report/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/deployment-report/meta/main.yml b/metron-deployment/roles/deployment-report/meta/main.yml deleted file mode 100644 index c3d807b..0000000 --- a/metron-deployment/roles/deployment-report/meta/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - ambari_gather_facts http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/deployment-report/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/deployment-report/tasks/main.yml b/metron-deployment/roles/deployment-report/tasks/main.yml deleted file mode 100644 index ac84ee8..0000000 --- a/metron-deployment/roles/deployment-report/tasks/main.yml +++ /dev/null @@ -1,37 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the 'License'); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an 'AS IS' BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Sanity check Metron web - local_action: wait_for host="{{ groups.web[0] }}" port=5000 timeout="{{ timeout_secs }}" - -- name: Sanity check Ambari web - local_action: wait_for host="{{ groups.ambari_master[0] }}" port="{{ ambari_port }}" timeout="{{ timeout_secs }}" - -- name: Known hosts groups - debug: var=groups - -- set_fact: - Success: - - "Apache Metron deployed successfully" - - " Metron @ http://{{ groups.web[0] }}:5000" - - " Ambari @ http://{{ groups.ambari_master[0] }}:{{ ambari_port }}" - - " Sensor Status @ http://{{ groups.sensors[0] }}:2812" - - " Zookeeper @ {{ zookeeper_url }}" - - " Kafka @ {{ kafka_broker_url }}" - - For additional information, see https://metron.apache.org/' - -- debug: var=Success http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/epel/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/epel/tasks/main.yml b/metron-deployment/roles/epel/tasks/main.yml deleted file mode 100644 index 0962e40..0000000 --- a/metron-deployment/roles/epel/tasks/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Install EPEL repository - yum: name=epel-release http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/defaults/main.yml b/metron-deployment/roles/fastcapa/defaults/main.yml deleted file mode 100644 index d586046..0000000 --- a/metron-deployment/roles/fastcapa/defaults/main.yml +++ /dev/null @@ -1,42 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -# dpdk -dpdk_home: "/usr/local/dpdk" -dpdk_version: "17.08" -dpdk_sdk: "/root/dpdk-{{ dpdk_version }}" -dpdk_src_url: "http://fast.dpdk.org/rel/dpdk-{{ dpdk_version }}.tar.xz" -dpdk_target: "x86_64-native-linuxapp-gcc" -num_huge_pages: 512 -extra_cflags: -g - -# fastcapa -fastcapa_work_dir: /root/fastcapa -fastcapa_build_dir: "{{ fastcapa_work_dir }}/build/app/" -fastcapa_prefix: /usr/local/bin -fastcapa_ld_library_path: /usr/local/lib -fastcapa_bin: fastcapa - -# fastcapa settings -fastcapa_portmask: 0x01 -fastcapa_kafka_config: /etc/fastcapa.conf -fastcapa_topic: pcap -fastcapa_rx_burst_size: 32 -fastcapa_tx_burst_size: 256 -fastcapa_nb_rx_desc: 1024 -fastcapa_nb_rx_queue: 1 -fastcapa_tx_ring_size: 2048 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/meta/main.yml b/metron-deployment/roles/fastcapa/meta/main.yml deleted file mode 100644 index d253e88..0000000 --- a/metron-deployment/roles/fastcapa/meta/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - librdkafka http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/tasks/debug.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/tasks/debug.yml b/metron-deployment/roles/fastcapa/tasks/debug.yml deleted file mode 100644 index 06f1526..0000000 --- a/metron-deployment/roles/fastcapa/tasks/debug.yml +++ /dev/null @@ -1,26 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - - name: Install debug utilities - yum: name=yum-utils - tags: - - debug - - - name: Install debug symbols - shell: debuginfo-install -y glibc glib2 zlib - tags: - - debug http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/tasks/dependencies.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/tasks/dependencies.yml b/metron-deployment/roles/fastcapa/tasks/dependencies.yml deleted file mode 100644 index 5955702..0000000 --- a/metron-deployment/roles/fastcapa/tasks/dependencies.yml +++ /dev/null @@ -1,39 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - - name: Install dependencies - yum: name={{ item }} - with_items: - - "@Development tools" - - pciutils - - net-tools - - glib2 - - glib2-devel - - git - - numactl-devel - - # - # install prerequisite packages and the latest kernel headers. need to - # ensure that the kernel headers match the current running kernel version. - # if this is not the case, the DPDK build process will fail - # - - name: Install latest kernel headers and source - yum: name={{ item }} state=latest - with_items: - - kernel - - kernel-devel - - kernel-headers http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/tasks/dpdk.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/tasks/dpdk.yml b/metron-deployment/roles/fastcapa/tasks/dpdk.yml deleted file mode 100644 index a936570..0000000 --- a/metron-deployment/roles/fastcapa/tasks/dpdk.yml +++ /dev/null @@ -1,59 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - - name: "Download DPDK version {{ dpdk_version }}" - unarchive: - src: "{{ dpdk_src_url }}" - dest: "/root" - creates: "{{ dpdk_sdk }}" - copy: no - - - name: "Configure DPDK for the target environment: {{ dpdk_target }}" - shell: "make config T={{ dpdk_target }} DESTDIR={{ dpdk_home }}" - args: - chdir: "{{ dpdk_sdk }}" - creates: "{{ dpdk_home }}" - - - name: "Turn on debug flags" - lineinfile: - dest: "{{ dpdk_sdk }}/config/common_linuxapp" - regexp: 'DEBUG=n' - line: 'DEBUG=y' - tags: - - debug - - - name: "Build DPDK for the target environment: {{ dpdk_target }}" - shell: "make install T={{ dpdk_target }} DESTDIR={{ dpdk_home }} EXTRA_CFLAGS={{ extra_cflags }}" - args: - chdir: "{{ dpdk_sdk }}" - creates: "{{ dpdk_home }}" - - - name: Load kernel modules to enable userspace IO - shell: "{{ item }}" - with_items: - - modprobe uio_pci_generic - - modprobe vfio-pci - - - name: Bind the device to the loaded kernel module(s) - shell: "{{ dpdk_home }}/sbin/dpdk-devbind --force --bind=uio_pci_generic {{ item }}" - with_items: "{{ dpdk_device }}" - - - name: Set useful environment variables - lineinfile: "dest=/root/.bash_profile line={{ item }}" - with_items: - - "export RTE_SDK={{ dpdk_sdk }}" - - "export RTE_TARGET={{ dpdk_target }}" http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/tasks/fastcapa.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/tasks/fastcapa.yml b/metron-deployment/roles/fastcapa/tasks/fastcapa.yml deleted file mode 100644 index b555668..0000000 --- a/metron-deployment/roles/fastcapa/tasks/fastcapa.yml +++ /dev/null @@ -1,46 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Distribute fastcapa - copy: src=../../../metron-sensors/fastcapa dest={{ fastcapa_work_dir | dirname }} mode=0755 - -- name: Build fastcapa - shell: "{{ item }}" - args: - chdir: "{{ fastcapa_work_dir }}" - with_items: - - make - environment: - RTE_SDK: "{{ dpdk_sdk }}" - RTE_TARGET: "{{ dpdk_target }}" - LD_LIBRARY_PATH: "{{ fastcapa_ld_library_path }}" - -- name: Install fastcapa - shell: "cp {{ fastcapa_build_dir }}/{{ fastcapa_bin }} {{ fastcapa_prefix }}" - args: - chdir: "{{ fastcapa_work_dir }}" - creates: "{{ fastcapa_prefix }}/{{ fastcapa_bin }}" - -- name: Deploy configuration - template: src=fastcapa.conf dest={{ fastcapa_kafka_config }} mode=0755 - -- name: Deploy service - template: src=fastcapa dest=/etc/init.d/ mode=0755 - -- name: Register the service with systemd - shell: systemctl enable fastcapa - when: ansible_distribution == "CentOS" and ansible_distribution_major_version == "7" http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/tasks/kernel.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/tasks/kernel.yml b/metron-deployment/roles/fastcapa/tasks/kernel.yml deleted file mode 100644 index cd4abe6..0000000 --- a/metron-deployment/roles/fastcapa/tasks/kernel.yml +++ /dev/null @@ -1,51 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# -# DPDK requires specific kernel boot parameters. set the params and reboot -# the host, if the actual params differ from what is expected. -# ---- - - set_fact: - expected_kernel_params: "default_hugepagesz=1G hugepagesz=1G hugepages={{ num_huge_pages }} iommu=pt intel_iommu=on" - - - name: Check kernel boot parameters - shell: "cat /proc/cmdline" - register: actual_kernel_params - - - name: Alter kernel boot parameters - lineinfile: - dest: /etc/default/grub - regexp: '^(GRUB_CMDLINE_LINUX=\"[^\"]+)\"$' - line: '\1 {{ expected_kernel_params }}"' - backrefs: yes - when: not expected_kernel_params in actual_kernel_params.stdout - - - name: Update grub with kernel boot parameters - shell: /sbin/grub2-mkconfig -o /boot/grub2/grub.cfg - when: not expected_kernel_params in actual_kernel_params.stdout - - - name: Restart for modified kernel params - command: shutdown -r now "modified kernel params" - async: 0 - poll: 0 - ignore_errors: true - when: not expected_kernel_params in actual_kernel_params.stdout - - - name: Wait for reboot of '{{ inventory_hostname }}' - local_action: wait_for host={{ inventory_hostname }} state=started port=22 timeout=300 delay=10 - become: false - when: not expected_kernel_params in actual_kernel_params.stdout http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/tasks/main.yml b/metron-deployment/roles/fastcapa/tasks/main.yml deleted file mode 100644 index b98d557..0000000 --- a/metron-deployment/roles/fastcapa/tasks/main.yml +++ /dev/null @@ -1,22 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - - include: dependencies.yml - - include: kernel.yml - - include: dpdk.yml - - include: fastcapa.yml - - include: debug.yml http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/templates/fastcapa ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/templates/fastcapa b/metron-deployment/roles/fastcapa/templates/fastcapa deleted file mode 100644 index f34c603..0000000 --- a/metron-deployment/roles/fastcapa/templates/fastcapa +++ /dev/null @@ -1,145 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# fastcapa daemon -# chkconfig: 345 20 80 -# description: Packet capture probe -# processname: fastcapa -# - -export LD_LIBRARY_PATH="{{ fastcapa_ld_library_path }}" - -NAME="fastcapa" -DESC="Metron network packet capture probe" -PIDFILE=/var/run/$NAME.pid -SCRIPTNAME=/etc/init.d/$NAME -LOGROOT=/var/log/fastcapa -DAEMONLOG=$LOGROOT/$NAME-stdout.log -DAEMONERR=$LOGROOT/$NAME-stderr.log -NOW=`date` -DAEMON_PATH="/root" - -PORT_MASK="{{ fastcapa_portmask }}" -KAFKA_TOPIC="{{ fastcapa_topic }}" -KAFKA_CONFIG="{{ fastcapa_kafka_config }}" -RX_BURST_SIZE="{{ fastcapa_rx_burst_size }}" -TX_BURST_SIZE="{{ fastcapa_tx_burst_size }}" -NB_RX_DESC="{{ fastcapa_nb_rx_desc }}" -NB_RX_QUEUE="{{ fastcapa_nb_rx_queue }}" -TX_RING_SIZE="{{ fastcapa_tx_ring_size }}" - -DAEMON="{{ fastcapa_prefix }}/{{ fastcapa_bin }}" -DAEMONOPTS+=" " -DAEMONOPTS+=" -- " -DAEMONOPTS+="-p $PORT_MASK " -DAEMONOPTS+="-t $KAFKA_TOPIC " -DAEMONOPTS+="-c $KAFKA_CONFIG " -DAEMONOPTS+="-b $RX_BURST_SIZE " -DAEMONOPTS+="-w $TX_BURST_SIZE " -DAEMONOPTS+="-d $NB_RX_DESC " -DAEMONOPTS+="-q $NB_RX_QUEUE " -DAEMONOPTS+="-x $TX_RING_SIZE " - -case "$1" in - start) - printf "%-50s" "Starting $NAME..." - echo "$NOW: Starting $NAME..." >> $DAEMONLOG - - mkdir -p $LOGROOT - touch $DAEMONLOG - touch $DAEMONERR - - cd $DAEMON_PATH - echo "$DAEMON $DAEMONOPTS >> $DAEMONLOG 2> $DAEMONERR" >> $DAEMONLOG - - if [ -f $PIDFILE ]; then - printf "%s\n" "Already running" - else - PID=`$DAEMON $DAEMONOPTS >> $DAEMONLOG 2> $DAEMONERR & echo $!` - if [ -z $PID ]; then - printf "%s\n" "Fail" - else - echo $PID > $PIDFILE - printf "%s\n" "Ok" - fi - fi - ;; - - status) - printf "%-50s" "Checking $NAME..." - if [ -f $PIDFILE ]; then - PID=`cat $PIDFILE` - if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then - printf "%s\n" "Process dead but pidfile exists" - else - echo "Running" - fi - else - printf "%s\n" "Service not running" - fi - ;; - - stop) - printf "%-50s" "Stopping $NAME" - PID=`cat $PIDFILE` - cd $DAEMON_PATH - if [ -f $PIDFILE ]; then - while sleep 1 - echo -n "." - kill -0 $PID >/dev/null 2>&1 - do - kill -SIGINT $PID - done - printf "%s\n" "Ok" - rm -f $PIDFILE - else - printf "%s\n" "pidfile not found" - fi - ;; - - restart) - $0 stop - $0 start - ;; - - tail) - tail -F $LOGROOT/* - ;; - - kill) - printf "%-50s" "Force killing $NAME" - PID=`cat $PIDFILE` - cd $DAEMON_PATH - if [ -f $PIDFILE ]; then - while sleep 1 - echo -n "." - kill -0 $PID >/dev/null 2>&1 - do - kill -SIGTERM $PID - done - printf "%s\n" "Ok" - rm -f $PIDFILE - else - printf "%s\n" "pidfile not found" - fi - ;; - - *) - echo "Usage: $0 {status|start|stop|restart|kill|tail}" - exit 1 -esac http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/fastcapa/templates/fastcapa.conf ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/fastcapa/templates/fastcapa.conf b/metron-deployment/roles/fastcapa/templates/fastcapa.conf deleted file mode 100644 index 7d9eae4..0000000 --- a/metron-deployment/roles/fastcapa/templates/fastcapa.conf +++ /dev/null @@ -1,67 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# -# kafka global settings -# -[kafka-global] - -# initial list of kafka brokers -metadata.broker.list = {{ kafka_broker_url }} - -# identifies the client to kafka -client.id = metron-fastcapa - -# max number of messages allowed on the producer queue -queue.buffering.max.messages = 1000000 - -# maximum time, in milliseconds, for buffering data on the producer queue -queue.buffering.max.ms = 3000 - -# compression codec = none, gzip or snappy -compression.codec = snappy - -# maximum number of messages batched in one MessageSet (increase for better compression) -batch.num.messages = 10 - -# max times to retry sending a failed message set -message.send.max.retries = 5 - -# backoff time before retrying a message send -retry.backoff.ms = 250 - -# how often statistics are emitted; 0 = never -statistics.interval.ms = 5000 - -# only provide delivery reports for failed messages -delivery.report.only.error = false - -# -# kafka topic settings -# -[kafka-topic] - -# broker acks { 1 = leader ack, 0 = no acks, -1 = in sync replica ack } -request.required.acks = 1 - -# local message timeout. This value is only enforced locally and limits the time a -# produced message waits for successful delivery. A time of 0 is infinite. -message.timeout.ms = 10000 - -# report offset of produced message back to application. The application must be -# use the dr_msg_cb to retrieve the offset from rd_kafka_message_t.offset -produce.offset.report = false http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/httplib2/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/httplib2/tasks/main.yml b/metron-deployment/roles/httplib2/tasks/main.yml deleted file mode 100644 index 5502cf4..0000000 --- a/metron-deployment/roles/httplib2/tasks/main.yml +++ /dev/null @@ -1,20 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Install python httplib2 dependency - pip: - name: httplib2 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/java_jdk/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/java_jdk/defaults/main.yml b/metron-deployment/roles/java_jdk/defaults/main.yml deleted file mode 100644 index 315bc0c..0000000 --- a/metron-deployment/roles/java_jdk/defaults/main.yml +++ /dev/null @@ -1,18 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -java_home: /usr/jdk64/jdk1.8.0_77 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/java_jdk/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/java_jdk/tasks/main.yml b/metron-deployment/roles/java_jdk/tasks/main.yml deleted file mode 100644 index 999b9c1..0000000 --- a/metron-deployment/roles/java_jdk/tasks/main.yml +++ /dev/null @@ -1,34 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Check for java at "{{ java_home }}" - stat: path="{{ java_home }}" - register: jdk_dir - -- name: Alternatives link for java - alternatives: name={{ item.name }} link={{ item.link }} path={{ item.path }} - with_items: - - { name: java, link: /usr/bin/java, path: "{{ java_home }}/bin/java" } - - { name: jar, link: /usr/bin/jar, path: "{{ java_home }}/bin/jar" } - when: jdk_dir.stat.exists - -- name: Install openjdk - yum: name={{item}} - with_items: - - java-1.8.0-openjdk - - java-1.8.0-openjdk-devel - when: not jdk_dir.stat.exists http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/kafka-broker/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/kafka-broker/defaults/main.yml b/metron-deployment/roles/kafka-broker/defaults/main.yml deleted file mode 100644 index b0f5a11..0000000 --- a/metron-deployment/roles/kafka-broker/defaults/main.yml +++ /dev/null @@ -1,18 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -hdp_repo_def: "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo" http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/kafka-broker/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/kafka-broker/meta/main.yml b/metron-deployment/roles/kafka-broker/meta/main.yml deleted file mode 100644 index 9587e79..0000000 --- a/metron-deployment/roles/kafka-broker/meta/main.yml +++ /dev/null @@ -1,18 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -dependencies: - - libselinux-python http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/kafka-broker/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/kafka-broker/tasks/main.yml b/metron-deployment/roles/kafka-broker/tasks/main.yml deleted file mode 100644 index db05cb0..0000000 --- a/metron-deployment/roles/kafka-broker/tasks/main.yml +++ /dev/null @@ -1,41 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Retrieve HDP repository definition - get_url: - url: "{{ hdp_repo_def }}" - dest: /etc/yum.repos.d/hdp.repo - mode: 0644 - -- name: Install kafka - yum: name={{item}} - with_items: - - java-1.8.0-openjdk - - kafka - - zookeeper-server - -- name: Create pid directories - file: path={{ item }} state=directory mode=0755 - with_items: - - /var/run/zookeeper - - /var/run/kafka - -- name: Start zookeeper - shell: /usr/hdp/current/zookeeper-server/bin/zookeeper-server start - -- name: Start kafka - shell: /usr/hdp/current/kafka-broker/bin/kafka start http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/kafka-client/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/kafka-client/defaults/main.yml b/metron-deployment/roles/kafka-client/defaults/main.yml deleted file mode 100644 index b0f5a11..0000000 --- a/metron-deployment/roles/kafka-client/defaults/main.yml +++ /dev/null @@ -1,18 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -hdp_repo_def: "http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.5.0.0/hdp.repo" http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/kafka-client/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/kafka-client/tasks/main.yml b/metron-deployment/roles/kafka-client/tasks/main.yml deleted file mode 100644 index 1674225..0000000 --- a/metron-deployment/roles/kafka-client/tasks/main.yml +++ /dev/null @@ -1,30 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - - name: Retrieve HDP repository definition - get_url: - url: "{{ hdp_repo_def }}" - dest: /etc/yum.repos.d/hdp.repo - mode: 0644 - - - name: Install kafka - yum: - name: kafka - register: result - until: result.rc == 0 - retries: 5 - delay: 10 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/librdkafka/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/librdkafka/defaults/main.yml b/metron-deployment/roles/librdkafka/defaults/main.yml deleted file mode 100644 index 063c22f..0000000 --- a/metron-deployment/roles/librdkafka/defaults/main.yml +++ /dev/null @@ -1,20 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -librdkafka_version: 0.9.4 -librdkafka_url: https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -librdkafka_home: /usr/local http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/librdkafka/tasks/dependencies.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/librdkafka/tasks/dependencies.yml b/metron-deployment/roles/librdkafka/tasks/dependencies.yml deleted file mode 100644 index 72ff907..0000000 --- a/metron-deployment/roles/librdkafka/tasks/dependencies.yml +++ /dev/null @@ -1,40 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Install prerequisites - yum: name={{ item }} - with_items: - - cmake - - make - - gcc - - gcc-c++ - - flex - - bison - - libpcap - - libpcap-devel - - openssl-devel - - python-devel - - swig - - zlib-devel - - perl - - cyrus-sasl - - cyrus-sasl-devel - - cyrus-sasl-gssapi - register: result - until: result.rc == 0 - retries: 5 - delay: 10 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/librdkafka/tasks/librdkafka.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/librdkafka/tasks/librdkafka.yml b/metron-deployment/roles/librdkafka/tasks/librdkafka.yml deleted file mode 100644 index a7971d6..0000000 --- a/metron-deployment/roles/librdkafka/tasks/librdkafka.yml +++ /dev/null @@ -1,39 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Download librdkafka - get_url: - url: "{{ librdkafka_url }}" - dest: "/tmp/librdkafka-{{ librdkafka_version }}.tar.gz" - -- name: Extract librdkafka tarball - unarchive: - src: "/tmp/librdkafka-{{ librdkafka_version }}.tar.gz" - dest: /tmp - copy: no - creates: "/tmp/librdkafka-{{ librdkafka_version }}" - -- name: Compile and install librdkafka - shell: "{{ item }}" - args: - chdir: "/tmp/librdkafka-{{ librdkafka_version }}" - creates: "{{ librdkafka_home }}/lib/librdkafka.so" - with_items: - - rm -rf build/ - - "./configure --prefix={{ librdkafka_home }} --enable-sasl" - - make - - make install http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/librdkafka/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/librdkafka/tasks/main.yml b/metron-deployment/roles/librdkafka/tasks/main.yml deleted file mode 100644 index 2144d7f..0000000 --- a/metron-deployment/roles/librdkafka/tasks/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- include: dependencies.yml -- include: librdkafka.yml http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/libselinux-python/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/libselinux-python/tasks/main.yml b/metron-deployment/roles/libselinux-python/tasks/main.yml deleted file mode 100644 index 78f5a27..0000000 --- a/metron-deployment/roles/libselinux-python/tasks/main.yml +++ /dev/null @@ -1,25 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Install libselinux-python - yum: - name: libselinux-python - state: installed - register: result - until: result.rc == 0 - retries: 5 - delay: 10 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/load_web_templates/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/load_web_templates/meta/main.yml b/metron-deployment/roles/load_web_templates/meta/main.yml deleted file mode 100644 index c3d807b..0000000 --- a/metron-deployment/roles/load_web_templates/meta/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - ambari_gather_facts http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/load_web_templates/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/load_web_templates/tasks/main.yml b/metron-deployment/roles/load_web_templates/tasks/main.yml deleted file mode 100644 index 3a91960..0000000 --- a/metron-deployment/roles/load_web_templates/tasks/main.yml +++ /dev/null @@ -1,24 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Load Kibana Dashboard - command: > - curl -s -w "%{http_code}" -u admin:admin -H "X-Requested-By: ambari" -X POST -d '{ "RequestInfo": { "context": "Install Kibana Dashboard from REST", "command": "LOAD_TEMPLATE"},"Requests/resource_filters": [{"service_name": "KIBANA","component_name": "KIBANA_MASTER","hosts" : "{{ kibana_hosts[0] }}"}]}' http://{{ groups.ambari_master[0] }}:{{ ambari_port }}/api/v1/clusters/{{ cluster_name }}/requests - args: - warn: off - register: result - failed_when: "result.rc != 0 or '202' not in result.stdout" http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron-builder/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron-builder/defaults/main.yml b/metron-deployment/roles/metron-builder/defaults/main.yml deleted file mode 100644 index 07f22f4..0000000 --- a/metron-deployment/roles/metron-builder/defaults/main.yml +++ /dev/null @@ -1,18 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -metron_build_dir: "{{ playbook_dir }}/../.." http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron-builder/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron-builder/tasks/main.yml b/metron-deployment/roles/metron-builder/tasks/main.yml deleted file mode 100644 index d5fb517..0000000 --- a/metron-deployment/roles/metron-builder/tasks/main.yml +++ /dev/null @@ -1,26 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Build Metron - local_action: shell cd {{ metron_build_dir }} && mvn clean package -DskipTests -T 2C -P HDP-2.5.0.0,mpack - become: false - run_once: true - -- name: Build Metron RPMs - local_action: shell cd {{ metron_build_dir }}/metron-deployment && mvn clean package -DskipTests -P build-rpms - become: false - run_once: true http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron-rpms/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron-rpms/defaults/main.yml b/metron-deployment/roles/metron-rpms/defaults/main.yml deleted file mode 100644 index 4a9d606..0000000 --- a/metron-deployment/roles/metron-rpms/defaults/main.yml +++ /dev/null @@ -1,18 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -metron_rpm_glob: "{{ playbook_dir }}/../packaging/docker/rpm-docker/target/RPMS/noarch/*.rpm" http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron-rpms/tasks/copy_rpms.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron-rpms/tasks/copy_rpms.yml b/metron-deployment/roles/metron-rpms/tasks/copy_rpms.yml deleted file mode 100644 index 3410b7f..0000000 --- a/metron-deployment/roles/metron-rpms/tasks/copy_rpms.yml +++ /dev/null @@ -1,26 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Copy Metron RPMs - copy: - src: "{{ item }}" - dest: /localrepo - owner: root - mode: 0755 - with_fileglob: - - "{{ metron_rpm_glob }}" - http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron-rpms/tasks/create_directory.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron-rpms/tasks/create_directory.yml b/metron-deployment/roles/metron-rpms/tasks/create_directory.yml deleted file mode 100644 index 0601625..0000000 --- a/metron-deployment/roles/metron-rpms/tasks/create_directory.yml +++ /dev/null @@ -1,28 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - -- name: Clean out localrepo - file: - state: absent - path: /localrepo - -- name: Create localrepo directory - file: - path: /localrepo - state: directory - mode: 0755 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron-rpms/tasks/create_repo.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron-rpms/tasks/create_repo.yml b/metron-deployment/roles/metron-rpms/tasks/create_repo.yml deleted file mode 100644 index e2ab00a..0000000 --- a/metron-deployment/roles/metron-rpms/tasks/create_repo.yml +++ /dev/null @@ -1,28 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Install createrepo - package: - name: createrepo - state: installed - -- name: Create local repo with new packages - shell: "createrepo /localrepo" - args: - creates: /localrepo/repodata - - http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron-rpms/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron-rpms/tasks/main.yml b/metron-deployment/roles/metron-rpms/tasks/main.yml deleted file mode 100644 index 9ebb452..0000000 --- a/metron-deployment/roles/metron-rpms/tasks/main.yml +++ /dev/null @@ -1,21 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - -- include: create_directory.yml -- include: copy_rpms.yml -- include: create_repo.yml http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/error_index.template ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/error_index.template b/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/error_index.template deleted file mode 100644 index 3bb4633..0000000 --- a/metron-deployment/roles/metron_elasticsearch_templates/files/es_templates/error_index.template +++ /dev/null @@ -1,57 +0,0 @@ -{ - "template": "error_index*", - "mappings": { - "error_doc": { - "_timestamp": { - "enabled": true - }, - "properties": { - "exception": { - "type": "string", - "index": "not_analyzed" - }, - "hostname": { - "type": "string", - "index": "not_analyzed" - }, - "stack": { - "type": "string", - "index": "not_analyzed" - }, - "timestamp": { - "type": "date", - "format": "epoch_millis" - }, - "message": { - "type": "string", - "index": "not_analyzed" - }, - "raw_message": { - "type": "string", - "index": "not_analyzed", - "ignore_above": 8191 - }, - "raw_message_bytes": { - "type": "binary", - "index": "no" - }, - "error_fields": { - "type": "string", - "index": "not_analyzed" - }, - "error_hash": { - "type": "string", - "index": "not_analyzed" - }, - "failed_sensor_type": { - "type": "string", - "index": "not_analyzed" - }, - "error_type": { - "type": "string", - "index": "not_analyzed" - } - } - } - } -} http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_pcapservice/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_pcapservice/defaults/main.yml b/metron-deployment/roles/metron_pcapservice/defaults/main.yml deleted file mode 100644 index c28938b..0000000 --- a/metron-deployment/roles/metron_pcapservice/defaults/main.yml +++ /dev/null @@ -1,29 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -metron_version: 0.4.3 -metron_directory: /usr/metron/{{ metron_version }} -pcapservice_jar_name: metron-api-{{ metron_version }}.jar -pcapservice_jar_src: "{{ playbook_dir }}/../../metron-platform/metron-api/target/{{ pcapservice_jar_name }}" -pcapservice_jar_dst: "{{ metron_directory }}/lib/{{ pcapservice_jar_name }}" -pcapservice_port: 8081 -hbase_config_path: "/etc/hbase/conf" -query_hdfs_path: "/tmp" -pcap_hdfs_path: "/apps/metron/pcap" -metron_pcapservice_logrotate_frequency: daily -metron_pcapservice_logrotate_retention: 30 - http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_pcapservice/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_pcapservice/meta/main.yml b/metron-deployment/roles/metron_pcapservice/meta/main.yml deleted file mode 100644 index ddf6aa9..0000000 --- a/metron-deployment/roles/metron_pcapservice/meta/main.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - java_jdk http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_pcapservice/tasks/config-hbase.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_pcapservice/tasks/config-hbase.yml b/metron-deployment/roles/metron_pcapservice/tasks/config-hbase.yml deleted file mode 100644 index b77c1ec..0000000 --- a/metron-deployment/roles/metron_pcapservice/tasks/config-hbase.yml +++ /dev/null @@ -1,26 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- - - name: Fetch hbase-site.xml - fetch: src=/etc/hbase/conf/hbase-site.xml dest=/tmp/hbase/conf/hbase-site.xml flat=yes - delegate_to: "{{ groups.ambari_slave[0] }}" - - - name: Create hbase conf directory - file: path=/etc/hbase/conf state=directory mode=0755 - - - name: Copy hbase-site.xml - copy: src=/tmp/hbase/conf/hbase-site.xml dest=/etc/hbase/conf/hbase-site.xml mode=0644 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_pcapservice/tasks/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_pcapservice/tasks/main.yml b/metron-deployment/roles/metron_pcapservice/tasks/main.yml deleted file mode 100644 index ed33354..0000000 --- a/metron-deployment/roles/metron_pcapservice/tasks/main.yml +++ /dev/null @@ -1,25 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- include: pcapservice.yml - -- name: Create Logrotate Script for metron_pcapservice - template: - src: "metron-pcapservice-logrotate.yml" - dest: "/etc/logrotate.d/metron-pcapservice" - mode: 0644 - http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_pcapservice/tasks/pcapservice.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_pcapservice/tasks/pcapservice.yml b/metron-deployment/roles/metron_pcapservice/tasks/pcapservice.yml deleted file mode 100644 index d0ac411..0000000 --- a/metron-deployment/roles/metron_pcapservice/tasks/pcapservice.yml +++ /dev/null @@ -1,30 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -- name: Create Metron streaming directories - file: path={{ metron_directory }}/{{ item.name }} state=directory mode=0755 - with_items: - - { name: 'lib'} - - { name: 'config'} - -- name: Copy Metron pcapservice jar - copy: - src: "{{ pcapservice_jar_src }}" - dest: "{{ pcapservice_jar_dst }}" - -- name: Install service script - template: src=pcapservice dest=/etc/init.d/pcapservice mode=0755 http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_pcapservice/templates/metron-pcapservice-logrotate.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_pcapservice/templates/metron-pcapservice-logrotate.yml b/metron-deployment/roles/metron_pcapservice/templates/metron-pcapservice-logrotate.yml deleted file mode 100644 index 0293ade..0000000 --- a/metron-deployment/roles/metron_pcapservice/templates/metron-pcapservice-logrotate.yml +++ /dev/null @@ -1,27 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -#Metron pcapService -/var/log/metron_pcapservice.log { - {{ metron_pcapservice_logrotate_frequency }} - rotate {{ metron_pcapservice_logrotate_retention }} - missingok - notifempty - copytruncate - compress -} - http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_pcapservice/templates/pcapservice ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_pcapservice/templates/pcapservice b/metron-deployment/roles/metron_pcapservice/templates/pcapservice deleted file mode 100644 index 054133d..0000000 --- a/metron-deployment/roles/metron_pcapservice/templates/pcapservice +++ /dev/null @@ -1,84 +0,0 @@ -#!/usr/bin/env bash -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# metron pcap service -# chkconfig: 345 20 80 -# description: Metron PCAP Service Daemon -# processname: pcapservice -# -NAME=pcapservice -DESC="Metron pcap service" -PIDFILE=/var/run/$NAME.pid -SCRIPTNAME=/etc/init.d/$NAME -LOGFILE="/var/log/metron_pcapservice.log" -EXTRA_ARGS="${@:2}" -DAEMON_PATH="/" -DAEMON="/usr/bin/yarn jar" -DAEMONOPTS="{{ pcapservice_jar_dst }} org.apache.metron.pcapservice.rest.PcapService -port {{ pcapservice_port }} -query_hdfs_path {{ query_hdfs_path }} -pcap_hdfs_path {{ pcap_hdfs_path }}" - -case "$1" in - start) - printf "%-50s" "Starting $NAME..." - - # kick-off the daemon - cd $DAEMON_PATH - PID=`$DAEMON $DAEMONOPTS >> $LOGFILE 2>&1 & echo $!` - if [ -z $PID ]; then - printf "%s\n" "Fail" - else - echo $PID > $PIDFILE - printf "%s\n" "Ok" - fi - ;; - - status) - printf "%-50s" "Checking $NAME..." - if [ -f $PIDFILE ]; then - PID=`cat $PIDFILE` - if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then - printf "%s\n" "Process dead but pidfile exists" - else - echo "Running" - fi - else - printf "%s\n" "Service not running" - fi - ;; - - stop) - printf "%-50s" "Stopping $NAME" - PID=`cat $PIDFILE` - cd $DAEMON_PATH - if [ -f $PIDFILE ]; then - kill -HUP $PID - printf "%s\n" "Ok" - rm -f $PIDFILE - else - printf "%s\n" "pidfile not found" - fi - ;; - - restart) - $0 stop - $0 start - ;; - - *) - echo "Usage: $0 {status|start|stop|restart}" - exit 1 -esac http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_streaming/defaults/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_streaming/defaults/main.yml b/metron-deployment/roles/metron_streaming/defaults/main.yml deleted file mode 100644 index b9b34fe..0000000 --- a/metron-deployment/roles/metron_streaming/defaults/main.yml +++ /dev/null @@ -1,98 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -metron_directory: /usr/metron/{{ metron_version }} -metron_solr_jar_name: metron-solr-{{ metron_version }}.jar -metron_elasticsearch_jar_name: metron-elasticsearch-{{ metron_version }}.jar -metron_pcap_jar_name: metron-pcap-backend-{{ metron_version }}.jar -metron_parsers_jar_name: metron-parsers-{{ metron_version }}.jar -metron_enrichment_jar_name: metron-enrichment-{{ metron_version }}-uber.jar -metron_profiler_jar_name: metron-profiler-{{ metron_version }}.jar - -# bundle names -metron_common_bundle_name: metron-common-{{ metron_version }}-archive.tar.gz -metron_pcap_bundle_name: metron-pcap-backend-{{ metron_version }}-archive.tar.gz -metron_data_management_bundle_name: metron-data-management-{{ metron_version }}-archive.tar.gz -metron_enrichment_bundle_name: metron-enrichment-{{ metron_version }}-archive.tar.gz -metron_indexing_bundle_name: metron-indexing-{{ metron_version }}-archive.tar.gz -metron_solr_bundle_name: metron-solr-{{ metron_version }}-archive.tar.gz -metron_elasticsearch_bundle_name: metron-elasticsearch-{{ metron_version }}-archive.tar.gz -metron_parsers_bundle_name: metron-parsers-{{ metron_version }}-archive.tar.gz -metron_maas_bundle_name: metron-maas-service-{{ metron_version }}-archive.tar.gz -metron_profiler_bundle_name: metron-profiler-{{ metron_version }}-archive.tar.gz -metron_profiler_client_bundle_name: metron-profiler-client-{{ metron_version }}-archive.tar.gz - -# bundle paths -metron_pcap_bundle_path: "{{ playbook_dir }}/../../metron-platform/metron-pcap-backend/target/{{ metron_pcap_bundle_name }}" -metron_common_bundle_path: "{{ playbook_dir }}/../../metron-platform/metron-common/target/{{ metron_common_bundle_name }}" -metron_data_management_bundle_path: "{{ playbook_dir }}/../../metron-platform/metron-data-management/target/{{ metron_data_management_bundle_name }}" -metron_enrichment_bundle_path: "{{ playbook_dir }}/../../metron-platform/metron-enrichment/target/{{ metron_enrichment_bundle_name }}" -metron_indexing_bundle_path: "{{ playbook_dir }}/../../metron-platform/metron-indexing/target/{{ metron_indexing_bundle_name }}" -metron_solr_bundle_path: "{{ playbook_dir }}/../../metron-platform/metron-solr/target/{{ metron_solr_bundle_name }}" -metron_elasticsearch_bundle_path: "{{ playbook_dir }}/../../metron-platform/metron-elasticsearch/target/{{ metron_elasticsearch_bundle_name }}" -metron_parsers_bundle_path: "{{ playbook_dir }}/../../metron-platform/metron-parsers/target/{{ metron_parsers_bundle_name }}" -metron_maas_bundle_path: "{{ playbook_dir }}/../../metron-analytics/metron-maas-service/target/{{ metron_maas_bundle_name }}" -metron_profiler_bundle_path: "{{ playbook_dir }}/../../metron-analytics/metron-profiler/target/{{ metron_profiler_bundle_name }}" -metron_profiler_client_bundle_path: "{{ playbook_dir }}/../../metron-analytics/metron-profiler-client/target/{{ metron_profiler_client_bundle_name }}" - - -# configuration paths -config_path: "{{ metron_directory }}/config" -zookeeper_config_path: "{{ config_path }}/zookeeper" -zookeeper_global_config_path: "{{ zookeeper_config_path }}/global.json" -metron_pcap_properties_config_path: "{{ metron_directory }}/config/pcap.properties" -metron_solr_properties_config_path: "{{ metron_directory }}/config/solr.properties" -metron_elasticsearch_properties_config_path: "{{ metron_directory }}/config/elasticsearch.properties" -metron_enrichment_properties_config_path: "{{ metron_directory }}/config/enrichment.properties" -metron_parsers_properties_config_path: "{{ metron_directory }}/config/parsers.properties" -metron_profiler_properties_config_path: "{{ metron_directory }}/config/profiler.properties" - -hbase_config_path: "/etc/hbase/conf" -hdfs_config_path: "/etc/hadoop/conf" -pcap_hdfs_path: "/apps/metron/pcap" -geo_hdfs_path: "/apps/metron/geo/default" - -threat_intel_bulk_load: True -threat_intel_bin: "{{ metron_directory }}/bin/flatfile_loader.sh" -threat_intel_work_dir: /tmp/ti_bulk -threat_intel_csv_filename: "threat_ip.csv" -threat_intel_csv_filepath: "{{ threat_intel_csv_filename }}" - -pycapa_topic: pcap -bro_topic: bro -yaf_topic: yaf -snort_topic: snort -enrichments_topic: enrichments - -hdfs_retention_days: 30 -hdfs_bro_purge_cronjob: "{{ metron_directory }}/bin/prune_hdfs_files.sh -f {{ hdfs_url }} -g '/apps/metron/indexing/indexed/bro_doc/*enrichment-*' -s $(date -d '{{ hdfs_retention_days }} days ago' +%m/%d/%Y) -n 1 >> /var/log/bro-purge/cron-hdfs-bro-purge.log 2>&1" -hdfs_yaf_purge_cronjob: "{{ metron_directory }}/bin/prune_hdfs_files.sh -f {{ hdfs_url }} -g '/apps/metron/indexing/indexed/yaf_doc/*enrichment-*' -s $(date -d '{{ hdfs_retention_days }} days ago' +%m/%d/%Y) -n 1 >> /var/log/yaf-purge/cron-hdfs-yaf-purge.log 2>&1" -hdfs_snort_purge_cronjob: "{{ metron_directory }}/bin/prune_hdfs_files.sh -f {{ hdfs_url }} -g '/apps/metron/indexing/indexed/snort_doc/*enrichment-*' -s $(date -d '{{ hdfs_retention_days }} days ago' +%m/%d/%Y) -n 1 >> /var/log/yaf-purge/cron-hdfs-snort-purge.log 2>&1" - -elasticsearch_config_path: /etc/elasticsearch -elasticsearch_cluster_name: metron -elasticsearch_transport_port: 9300 - -es_retention_days: 30 -es_bro_purge_cronjob: "{{ metron_directory }}/bin/prune_elasticsearch_indices.sh -z {{ zookeeper_url }} -p bro_index_ -s $(date -d '{{ es_retention_days }} days ago' +%m/%d/%Y) -n 1 >> /var/log/bro-purge/cron-es-bro-purge.log 2>&1" -es_yaf_purge_cronjob: "{{ metron_directory }}/bin/prune_elasticsearch_indices.sh -z {{ zookeeper_url }} -p yaf_index_ -s $(date -d '{{ es_retention_days }} days ago' +%m/%d/%Y) -n 1 >> /var/log/yaf-purge/cron-es-yaf-purge.log 2>&1" -es_snort_purge_cronjob: "{{ metron_directory }}/bin/prune_elasticsearch_indices.sh -z {{ zookeeper_url }} -p yaf_index_ -s $(date -d '{{ es_retention_days }} days ago' +%m/%d/%Y) -n 1 >> /var/log/snort-purge/cron-es-snort-purge.log 2>&1" - -metron_hdfs_output_dir: "/apps/metron" -metron_hdfs_rotation_policy: org.apache.storm.hdfs.bolt.rotation.TimedRotationPolicy -metron_hdfs_rotation_policy_count: 1 -metron_hdfs_rotation_policy_units: DAYS http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_streaming/files/extractor.json ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_streaming/files/extractor.json b/metron-deployment/roles/metron_streaming/files/extractor.json deleted file mode 100644 index 545202a..0000000 --- a/metron-deployment/roles/metron_streaming/files/extractor.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "config": { - "columns": { - "ip": 0 - }, - "indicator_column": "ip", - "type" : "malicious_ip", - "separator": "," - }, - "extractor": "CSV" -} - http://git-wip-us.apache.org/repos/asf/metron/blob/6f267991/metron-deployment/roles/metron_streaming/meta/main.yml ---------------------------------------------------------------------- diff --git a/metron-deployment/roles/metron_streaming/meta/main.yml b/metron-deployment/roles/metron_streaming/meta/main.yml deleted file mode 100644 index de2f386..0000000 --- a/metron-deployment/roles/metron_streaming/meta/main.yml +++ /dev/null @@ -1,21 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# ---- -dependencies: - - ambari_gather_facts - - java_jdk - - libselinux-python