metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickal...@apache.org
Subject metron-bro-plugin-kafka git commit: METRON-1304 Allow metron-bro-plugin-kafka to include or exclude logs (JonZeolla via nickwallen) closes apache/metron-bro-plugin-kafka#2
Date Sat, 10 Nov 2018 17:06:08 GMT
Repository: metron-bro-plugin-kafka
Updated Branches:
  refs/heads/master 279a2970b -> 772788d1a


METRON-1304 Allow metron-bro-plugin-kafka to include or exclude logs (JonZeolla via nickwallen)
closes apache/metron-bro-plugin-kafka#2


Project: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/commit/772788d1
Tree: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/tree/772788d1
Diff: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/diff/772788d1

Branch: refs/heads/master
Commit: 772788d1ab97a37abe8f8356c03b01c57aab4a8a
Parents: 279a297
Author: JonZeolla <zeolla@gmail.com>
Authored: Sat Nov 10 12:05:43 2018 -0500
Committer: nickallen <nickallen@apache.org>
Committed: Sat Nov 10 12:05:43 2018 -0500

----------------------------------------------------------------------
 README.md                                       | 81 ++++++++++++++++----
 scripts/Apache/Kafka/__load__.bro               |  2 +
 scripts/Apache/Kafka/logs-to-kafka.bro          | 45 +++++++----
 scripts/__load__.bro                            |  2 +
 scripts/init.bro                                | 47 +++++++++---
 tests/Baseline/kafka.l2s-l2e-no-overlap/output  |  4 +
 tests/Baseline/kafka.l2s-set-l2e-set/output     |  3 +
 tests/Baseline/kafka.l2s-set-l2e-unset/output   |  3 +
 tests/Baseline/kafka.l2s-unset-l2e-set/output   |  3 +
 tests/Baseline/kafka.l2s-unset-l2e-unset/output |  3 +
 .../kafka.send-all-active-logs-l2e-set/output   |  7 ++
 .../kafka.send-all-active-logs-l2e-unset/output |  7 ++
 .../output                                      |  7 ++
 .../output                                      |  7 ++
 tests/kafka/l2s-l2e-no-overlap.bro              | 29 +++++++
 tests/kafka/l2s-set-l2e-set.bro                 | 28 +++++++
 tests/kafka/l2s-set-l2e-unset.bro               | 27 +++++++
 tests/kafka/l2s-unset-l2e-set.bro               | 27 +++++++
 tests/kafka/l2s-unset-l2e-unset.bro             | 25 ++++++
 tests/kafka/send-all-active-logs-l2e-set.bro    | 32 ++++++++
 tests/kafka/send-all-active-logs-l2e-unset.bro  | 31 ++++++++
 .../send-all-active-logs-l2s-set-l2e-set.bro    | 33 ++++++++
 .../send-all-active-logs-l2s-set-l2e-unset.bro  | 32 ++++++++
 23 files changed, 447 insertions(+), 38 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 90d8444..ae6b260 100644
--- a/README.md
+++ b/README.md
@@ -48,7 +48,7 @@ The following examples highlight different ways that the plugin can be used.
 Si
 The goal in this example is to send all HTTP and DNS records to a Kafka topic named `bro`.
  * Any configuration value accepted by librdkafka can be added to the `kafka_conf` configuration
table.  
  * By defining `topic_name` all records will be sent to the same Kafka topic.
- * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent. 
+ * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent.
 ```
 @load packages/metron-bro-plugin-kafka/Apache/Kafka
 redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
@@ -60,6 +60,32 @@ redef Kafka::kafka_conf = table(
 
 ### Example 2
 
+This plugin has the ability send all active logs to kafka with the following configuration.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::send_all_active_logs = T;
+redef Kafka::kafka_conf = table(
+    ["metadata.broker.list"] = "localhost:9092"
+);
+```
+
+### Example 3
+
+You can also specify a blacklist of bro logs to ensure they aren't being sent to kafka regardless
of the `Kafka::send_all_active_logs` and `Kafka::logs_to_send` configurations.  In this example,
we will send all of the enabled logs except for the Conn log.
+
+```
+@load packages/metron-bro-plugin-kafka/Apache/Kafka
+redef Kafka::send_all_active_logs = T;
+redef Kafka::logs_to_exclude = set(Conn::LOG);
+redef Kafka::topic_name = "bro";
+redef Kafka::kafka_conf = table(
+    ["metadata.broker.list"] = "localhost:9092"
+);
+```
+
+### Example 4
+
 It is also possible to send each log stream to a uniquely named topic.  The goal in this
example is to send all HTTP records to a Kafka topic named `http` and all DNS records to a
separate Kafka topic named `dns`.
  * The `topic_name` value must be set to an empty string.
  * The `$path` value of Bro's Log Writer mechanism is used to define the topic name.
@@ -97,7 +123,7 @@ event bro_init()
 }
 ```
 
-### Example 3
+### Example 5
 
 You may want to configure bro to filter log messages with certain characteristics from being
sent to your kafka topics.  For instance, Metron currently doesn't support IPv6 source or
destination IPs in the default enrichments, so it may be helpful to filter those log messages
from being sent to kafka (although there are [multiple ways](#notes) to approach this).  In
this example we will do that that, and are assuming a somewhat standard bro kafka plugin configuration,
such that:
  * All bro logs are sent to the `bro` topic, by configuring `Kafka::topic_name`.
@@ -150,18 +176,29 @@ event bro_init() &priority=-5
 
 ## Settings
 
-### `kafka_conf`
+### `logs_to_send`
 
-The global configuration settings for Kafka.  These values are passed through
-directly to librdkafka.  Any valid librdkafka settings can be defined in this
-table.  The full set of valid librdkafka settings are available
-[here](https://github.com/edenhill/librdkafka/blob/v0.9.4/CONFIGURATION.md).
+A set of logs to send to kafka.
 
 ```
-redef Kafka::kafka_conf = table(
-    ["metadata.broker.list"] = "localhost:9092",
-    ["client.id"] = "bro"
-);
+redef Kafka::logs_to_send = set(Conn::LOG, DHCP::LOG);
+```
+
+### `send_all_active_logs`
+
+If true, all active logs will be sent to kafka other than those specified in
+`logs_to_exclude`.
+
+```
+redef Kafka::send_all_active_logs = T;
+```
+
+### `logs_to_exclude`
+
+A set of logs to exclude from being sent to kafka.
+
+```
+redef Kafka::logs_to_exclude = set(Conn::LOG, DNS::LOG);
 ```
 
 ### `topic_name`
@@ -172,13 +209,18 @@ The name of the topic in Kafka where all Bro logs will be sent to.
 redef Kafka::topic_name = "bro";
 ```
 
-### `max_wait_on_shutdown`
+### `kafka_conf`
 
-The maximum number of milliseconds that the plugin will wait for any backlog of
-queued messages to be sent to Kafka before forced shutdown.
+The global configuration settings for Kafka.  These values are passed through
+directly to librdkafka.  Any valid librdkafka settings can be defined in this
+table.  The full set of valid librdkafka settings are available
+[here](https://github.com/edenhill/librdkafka/blob/v0.9.4/CONFIGURATION.md).
 
 ```
-redef Kafka::max_wait_on_shutdown = 3000;
+redef Kafka::kafka_conf = table(
+    ["metadata.broker.list"] = "localhost:9092",
+    ["client.id"] = "bro"
+);
 ```
 
 ### `tag_json`
@@ -199,6 +241,15 @@ options are `JSON::TS_MILLIS` and `JSON::TS_ISO8601`.
 redef Kafka::json_timestamps = JSON::TS_ISO8601;
 ```
 
+### `max_wait_on_shutdown`
+
+The maximum number of milliseconds that the plugin will wait for any backlog of
+queued messages to be sent to Kafka before forced shutdown.
+
+```
+redef Kafka::max_wait_on_shutdown = 3000;
+```
+
 ### `debug`
 
 A comma separated list of debug contexts in librdkafka which you want to

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/scripts/Apache/Kafka/__load__.bro
----------------------------------------------------------------------
diff --git a/scripts/Apache/Kafka/__load__.bro b/scripts/Apache/Kafka/__load__.bro
index 2a1efa8..f5a85d6 100644
--- a/scripts/Apache/Kafka/__load__.bro
+++ b/scripts/Apache/Kafka/__load__.bro
@@ -14,6 +14,8 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
+
+#
 # This is loaded when a user activates the plugin. Include scripts here that should be
 # loaded automatically at that point.
 #

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/scripts/Apache/Kafka/logs-to-kafka.bro
----------------------------------------------------------------------
diff --git a/scripts/Apache/Kafka/logs-to-kafka.bro b/scripts/Apache/Kafka/logs-to-kafka.bro
index 8f12932..24d88a6 100644
--- a/scripts/Apache/Kafka/logs-to-kafka.bro
+++ b/scripts/Apache/Kafka/logs-to-kafka.bro
@@ -14,23 +14,40 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
-##! load this script to enable log output to kafka
+
+##! Load this script to enable log output to kafka
 
 module Kafka;
 
-event bro_init() &priority=-5
+function send_to_kafka(id: Log::ID): bool
 {
-	for (stream_id in Log::active_streams)
-	{
-		if (stream_id in Kafka::logs_to_send)
-		{
-			local filter: Log::Filter = [
-				$name = fmt("kafka-%s", stream_id),
-				$writer = Log::WRITER_KAFKAWRITER,
-				$config = table(["stream_id"] = fmt("%s", stream_id))
-			];
+        if (|logs_to_send| == 0 && send_all_active_logs == F)
+                # Send nothing unless it's explicitly set to send
+                return F;
+        else if (id in logs_to_exclude ||
+                (|logs_to_send| > 0 && id !in logs_to_send && send_all_active_logs
== F))
+                # Don't send logs in the exclusion set
+                return F;
+        else
+		# If send_all_active_logs is True, send all logs except those
+		# in the exclusion set.  Otherwise, send only the logs that are
+		# in the inclusion set, but not the exclusions set
+                return T;
+}
 
-			Log::add_filter(stream_id, filter);
-		}
-	}
+event bro_init() &priority=-10
+{
+        for (stream_id in Log::active_streams)
+        {
+                if (send_to_kafka(stream_id))
+                {
+                        local filter: Log::Filter = [
+                                $name = fmt("kafka-%s", stream_id),
+                                $writer = Log::WRITER_KAFKAWRITER,
+                                $config = table(["stream_id"] = fmt("%s", stream_id))
+                        ];
+        
+                        Log::add_filter(stream_id, filter);
+                }
+        }
 }

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/scripts/__load__.bro
----------------------------------------------------------------------
diff --git a/scripts/__load__.bro b/scripts/__load__.bro
index fee9549..e3db306 100644
--- a/scripts/__load__.bro
+++ b/scripts/__load__.bro
@@ -14,6 +14,8 @@
 #  See the License for the specific language governing permissions and
 #  limitations under the License.
 #
+
+#
 # This is loaded unconditionally at Bro startup. Include scripts here that should
 # always be loaded.
 #

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/scripts/init.bro
----------------------------------------------------------------------
diff --git a/scripts/init.bro b/scripts/init.bro
index ad9f0a1..08d46cd 100644
--- a/scripts/init.bro
+++ b/scripts/init.bro
@@ -18,13 +18,42 @@
 module Kafka;
 
 export {
-  const logs_to_send: set[Log::ID] &redef;
-  const topic_name: string = "bro" &redef;
-  const max_wait_on_shutdown: count = 3000 &redef;
-  const tag_json: bool = F &redef;
-  const json_timestamps: JSON::TimestampFormat = JSON::TS_EPOCH &redef;
-  const kafka_conf: table[string] of string = table(
-    ["metadata.broker.list"] = "localhost:9092"
-  ) &redef;
-  const debug: string = "" &redef;
+        ## Send all active logs to kafka except for those that are explicitly
+        ## excluded via logs_to_exclude.
+        ##
+        ## Example:  redef Kafka::send_all_active_logs = T;
+        const send_all_active_logs: bool = F &redef;
+
+        ## Specify which :bro:type:`Log::ID` to send to kafka.
+        ##
+        ## Example:  redef Kafka::logs_to_send = set(Conn::Log, DNS::LOG);
+        const logs_to_send: set[Log::ID] &redef;
+
+        ## Specify which :bro:type:`Log::ID` to exclude from being sent to kafka.
+        ##
+        ## Example:  redef Kafka::logs_to_exclude = set(SSH::LOG);
+        const logs_to_exclude: set[Log::ID] &redef;
+
+        ## Specify a different timestamp format.
+        ##
+        ## Example:  redef Kafka::json_timestamps = JSON::TS_ISO8601;
+        const json_timestamps: JSON::TimestampFormat = JSON::TS_EPOCH &redef;
+
+        ## Destination kafka topic name
+        const topic_name: string = "bro" &redef;
+
+        ## Maximum wait on shutdown in milliseconds
+        const max_wait_on_shutdown: count = 3000 &redef;
+
+        ## Whether or not to tag JSON with a log stream identifier
+        const tag_json: bool = F &redef;
+
+        ## Any additional configs to pass to librdkafka
+        const kafka_conf: table[string] of string = table(
+                ["metadata.broker.list"] = "localhost:9092"
+        ) &redef;
+
+        ## A comma separated list of librdkafka debug contexts
+        const debug: string = "" &redef;
 }
+

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.l2s-l2e-no-overlap/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.l2s-l2e-no-overlap/output b/tests/Baseline/kafka.l2s-l2e-no-overlap/output
new file mode 100644
index 0000000..405a6f3
--- /dev/null
+++ b/tests/Baseline/kafka.l2s-l2e-no-overlap/output
@@ -0,0 +1,4 @@
+T
+T
+F
+F

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.l2s-set-l2e-set/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.l2s-set-l2e-set/output b/tests/Baseline/kafka.l2s-set-l2e-set/output
new file mode 100644
index 0000000..3ea3c39
--- /dev/null
+++ b/tests/Baseline/kafka.l2s-set-l2e-set/output
@@ -0,0 +1,3 @@
+T
+F
+F

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.l2s-set-l2e-unset/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.l2s-set-l2e-unset/output b/tests/Baseline/kafka.l2s-set-l2e-unset/output
new file mode 100644
index 0000000..3a37b0c
--- /dev/null
+++ b/tests/Baseline/kafka.l2s-set-l2e-unset/output
@@ -0,0 +1,3 @@
+T
+T
+F

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.l2s-unset-l2e-set/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.l2s-unset-l2e-set/output b/tests/Baseline/kafka.l2s-unset-l2e-set/output
new file mode 100644
index 0000000..c5d6f61
--- /dev/null
+++ b/tests/Baseline/kafka.l2s-unset-l2e-set/output
@@ -0,0 +1,3 @@
+F
+F
+F

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.l2s-unset-l2e-unset/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.l2s-unset-l2e-unset/output b/tests/Baseline/kafka.l2s-unset-l2e-unset/output
new file mode 100644
index 0000000..c5d6f61
--- /dev/null
+++ b/tests/Baseline/kafka.l2s-unset-l2e-unset/output
@@ -0,0 +1,3 @@
+F
+F
+F

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.send-all-active-logs-l2e-set/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.send-all-active-logs-l2e-set/output b/tests/Baseline/kafka.send-all-active-logs-l2e-set/output
new file mode 100644
index 0000000..3fc91e9
--- /dev/null
+++ b/tests/Baseline/kafka.send-all-active-logs-l2e-set/output
@@ -0,0 +1,7 @@
+T
+T
+F
+F
+T
+F
+T

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.send-all-active-logs-l2e-unset/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.send-all-active-logs-l2e-unset/output b/tests/Baseline/kafka.send-all-active-logs-l2e-unset/output
new file mode 100644
index 0000000..7f6f673
--- /dev/null
+++ b/tests/Baseline/kafka.send-all-active-logs-l2e-unset/output
@@ -0,0 +1,7 @@
+T
+T
+T
+T
+T
+T
+T

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.send-all-active-logs-l2s-set-l2e-set/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.send-all-active-logs-l2s-set-l2e-set/output b/tests/Baseline/kafka.send-all-active-logs-l2s-set-l2e-set/output
new file mode 100644
index 0000000..6ca333d
--- /dev/null
+++ b/tests/Baseline/kafka.send-all-active-logs-l2s-set-l2e-set/output
@@ -0,0 +1,7 @@
+F
+T
+T
+T
+T
+F
+T

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/Baseline/kafka.send-all-active-logs-l2s-set-l2e-unset/output
----------------------------------------------------------------------
diff --git a/tests/Baseline/kafka.send-all-active-logs-l2s-set-l2e-unset/output b/tests/Baseline/kafka.send-all-active-logs-l2s-set-l2e-unset/output
new file mode 100644
index 0000000..7f6f673
--- /dev/null
+++ b/tests/Baseline/kafka.send-all-active-logs-l2s-set-l2e-unset/output
@@ -0,0 +1,7 @@
+T
+T
+T
+T
+T
+T
+T

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/l2s-l2e-no-overlap.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/l2s-l2e-no-overlap.bro b/tests/kafka/l2s-l2e-no-overlap.bro
new file mode 100644
index 0000000..a0ede70
--- /dev/null
+++ b/tests/kafka/l2s-l2e-no-overlap.bro
@@ -0,0 +1,29 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+redef logs_to_send = set(HTTP::LOG, DHCP::LOG);
+redef logs_to_exclude = set(Conn::LOG, DNS::LOG);
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(DHCP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/l2s-set-l2e-set.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/l2s-set-l2e-set.bro b/tests/kafka/l2s-set-l2e-set.bro
new file mode 100644
index 0000000..a13c68d
--- /dev/null
+++ b/tests/kafka/l2s-set-l2e-set.bro
@@ -0,0 +1,28 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+redef logs_to_send = set(HTTP::LOG, Conn::LOG);
+redef logs_to_exclude = set(Conn::LOG, DNS::LOG);
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/l2s-set-l2e-unset.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/l2s-set-l2e-unset.bro b/tests/kafka/l2s-set-l2e-unset.bro
new file mode 100644
index 0000000..439f578
--- /dev/null
+++ b/tests/kafka/l2s-set-l2e-unset.bro
@@ -0,0 +1,27 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+redef logs_to_send = set(HTTP::LOG, Conn::LOG);
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/l2s-unset-l2e-set.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/l2s-unset-l2e-set.bro b/tests/kafka/l2s-unset-l2e-set.bro
new file mode 100644
index 0000000..3898e3a
--- /dev/null
+++ b/tests/kafka/l2s-unset-l2e-set.bro
@@ -0,0 +1,27 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+redef logs_to_exclude = set(Conn::LOG, DNS::LOG);
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/l2s-unset-l2e-unset.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/l2s-unset-l2e-unset.bro b/tests/kafka/l2s-unset-l2e-unset.bro
new file mode 100644
index 0000000..7fd6b9f
--- /dev/null
+++ b/tests/kafka/l2s-unset-l2e-unset.bro
@@ -0,0 +1,25 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/send-all-active-logs-l2e-set.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/send-all-active-logs-l2e-set.bro b/tests/kafka/send-all-active-logs-l2e-set.bro
new file mode 100644
index 0000000..9019702
--- /dev/null
+++ b/tests/kafka/send-all-active-logs-l2e-set.bro
@@ -0,0 +1,32 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+redef send_all_active_logs = T;
+redef logs_to_exclude = set(Conn::LOG, DNS::LOG, SSL::LOG);
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(DHCP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);
+print send_to_kafka(SMTP::LOG);
+print send_to_kafka(SSL::LOG);
+print send_to_kafka(Files::LOG);

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/send-all-active-logs-l2e-unset.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/send-all-active-logs-l2e-unset.bro b/tests/kafka/send-all-active-logs-l2e-unset.bro
new file mode 100644
index 0000000..afef422
--- /dev/null
+++ b/tests/kafka/send-all-active-logs-l2e-unset.bro
@@ -0,0 +1,31 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+redef send_all_active_logs = T;
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(DHCP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);
+print send_to_kafka(SMTP::LOG);
+print send_to_kafka(SSL::LOG);
+print send_to_kafka(Files::LOG);

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/send-all-active-logs-l2s-set-l2e-set.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/send-all-active-logs-l2s-set-l2e-set.bro b/tests/kafka/send-all-active-logs-l2s-set-l2e-set.bro
new file mode 100644
index 0000000..6d223e2
--- /dev/null
+++ b/tests/kafka/send-all-active-logs-l2s-set-l2e-set.bro
@@ -0,0 +1,33 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+redef send_all_active_logs = T;
+redef logs_to_send = set(HTTP::LOG, Conn::LOG);
+redef logs_to_exclude = set(HTTP::LOG, SSL::LOG);
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(DHCP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);
+print send_to_kafka(SMTP::LOG);
+print send_to_kafka(SSL::LOG);
+print send_to_kafka(Files::LOG);

http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/tests/kafka/send-all-active-logs-l2s-set-l2e-unset.bro
----------------------------------------------------------------------
diff --git a/tests/kafka/send-all-active-logs-l2s-set-l2e-unset.bro b/tests/kafka/send-all-active-logs-l2s-set-l2e-unset.bro
new file mode 100644
index 0000000..cc7788d
--- /dev/null
+++ b/tests/kafka/send-all-active-logs-l2s-set-l2e-unset.bro
@@ -0,0 +1,32 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+# @TEST-EXEC: bro ../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+redef send_all_active_logs = T;
+redef logs_to_send = set(HTTP::LOG, Conn::LOG, SSL::LOG);
+
+print send_to_kafka(HTTP::LOG);
+print send_to_kafka(DHCP::LOG);
+print send_to_kafka(Conn::LOG);
+print send_to_kafka(DNS::LOG);
+print send_to_kafka(SMTP::LOG);
+print send_to_kafka(SSL::LOG);
+print send_to_kafka(Files::LOG);


Mime
View raw message