From commits-return-3502-archive-asf-public=cust-asf.ponee.io@metron.apache.org Wed Aug 1 18:15:39 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 50118180634 for ; Wed, 1 Aug 2018 18:15:39 +0200 (CEST) Received: (qmail 43671 invoked by uid 500); 1 Aug 2018 16:15:38 -0000 Mailing-List: contact commits-help@metron.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.apache.org Delivered-To: mailing list commits@metron.apache.org Received: (qmail 43662 invoked by uid 99); 1 Aug 2018 16:15:38 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 01 Aug 2018 16:15:38 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 3D3FBDFAB3; Wed, 1 Aug 2018 16:15:38 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: nickallen@apache.org To: commits@metron.apache.org Message-Id: <19a4bbfc1d1746b6b1d200fd5aff4869@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: metron git commit: METRON-1701 Update General notes on the installation of Pycapa on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136 Date: Wed, 1 Aug 2018 16:15:38 +0000 (UTC) Repository: metron Updated Branches: refs/heads/master a1e25a4bf -> 1b2bdff53 METRON-1701 Update General notes on the installation of Pycapa on Kerberized cluster (MohanDV via nickwallen) closes apache/metron#1136 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/1b2bdff5 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/1b2bdff5 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/1b2bdff5 Branch: refs/heads/master Commit: 1b2bdff53106694a9223ecc97f7ce9d0a78b4b09 Parents: a1e25a4 Author: MohanDV Authored: Wed Aug 1 12:13:19 2018 -0400 Committer: nickallen Committed: Wed Aug 1 12:13:19 2018 -0400 ---------------------------------------------------------------------- metron-sensors/pycapa/README.md | 47 ++++++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 18 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/1b2bdff5/metron-sensors/pycapa/README.md ---------------------------------------------------------------------- diff --git a/metron-sensors/pycapa/README.md b/metron-sensors/pycapa/README.md index 7e688a8..9f01d33 100644 --- a/metron-sensors/pycapa/README.md +++ b/metron-sensors/pycapa/README.md @@ -34,9 +34,9 @@ Pycapa performs network packet capture, both off-the-wire and from a Kafka topic Installation ============ -General notes on the installation of Pycapa. +General notes on the installation of Pycapa. * Python 2.7 is required. -* The following package dependencies are required and can be installed automatically with `pip`. +* The following package dependencies are required and can be installed automatically with `pip`. The requirements are installed as part of step 4 * [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python) * [pcapy](https://github.com/CoreSecurity/pcapy) * These instructions can be used directly on CentOS 7+. @@ -54,8 +54,8 @@ General notes on the installation of Pycapa. ``` export PREFIX=/usr - wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz - cd librdkafka-0.9.4/ + wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz + cd librdkafka-0.11.5/ ./configure --prefix=$PREFIX make make install @@ -231,18 +231,24 @@ Capturing on 'Standard input' ### Kerberos -The probe can be used in a Kerberized environment. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment. +The probe can be used in a Kerberized environment. The Python client README (https://github.com/confluentinc/confluent-kafka-python) has an important note for Kerberos case that the pre-built Linux wheels do NOT contain SASL Kerberos support. You will need to use the non-binary wheel to install confluent-kafka-python and build/install librdkafka separately. Follow these additional steps to use Pycapa with Kerberos. The following assumptions have been made. These may need altered to fit your environment. * The Kafka broker is at `kafka1:6667` * Zookeeper is at `zookeeper1:2181` * The Kafka security protocol is `SASL_PLAINTEXT` * The keytab used is located at `/etc/security/keytabs/metron.headless.keytab` * The service principal is `metron@EXAMPLE.COM` + + +1. If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. + ``` + yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi + ``` 1. Build Librdkafka with SASL support (` --enable-sasl`) and install at your chosen $PREFIX. ``` - wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O - | tar -xz - cd librdkafka-0.9.4/ + wget https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz -O - | tar -xz + cd librdkafka-0.11.5/ ./configure --prefix=$PREFIX --enable-sasl make make install @@ -250,14 +256,19 @@ The probe can be used in a Kerberized environment. Follow these additional step 1. Validate Librdkafka does indeed support SASL. Run the following command and ensure that `sasl` is returned as a built-in feature. ``` - $ examples/rdkafka_example -X builtin.features - builtin.features = gzip,snappy,ssl,sasl,regex - ``` - - If it is not, ensure that you have `libsasl` or `libsasl2` installed. On CentOS, this can be installed with the following command. - ``` - yum install -y cyrus-sasl cyrus-sasl-devel cyrus-sasl-gssapi + $ examples/rdkafka_example -X builtin.features + builtin.features = gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins ``` +1. The source install of confluent-kafka. + + If you have already installed, remove the binary wheel python client first, repeat until it says no longer installed + ``` + pip uninstall -y confluent-kafka + ``` + + ``` + pip install --no-binary :all: confluent-kafka + ``` 1. Grant access to your Kafka topic. In this example the topic is simply named `pcap`. ``` @@ -279,8 +290,8 @@ The probe can be used in a Kerberized environment. Follow these additional step * `security.protocol` * `sasl.kerberos.keytab` * `sasl.kerberos.principal` - - ``` + + ``` $ pycapa --producer \ --interface eth0 \ --kafka-broker kafka1:6667 \ @@ -292,8 +303,8 @@ The probe can be used in a Kerberized environment. Follow these additional step INFO:root:Starting packet capture INFO:root:Waiting for '1' message(s) to flush INFO:root:'10' packet(s) in, '10' packet(s) out - ``` - + ``` + FAQs ====