metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From l...@apache.org
Subject metron git commit: METRON-1220 Create documentation around alert nested field (justinleet) closes apache/metron#780
Date Fri, 06 Oct 2017 13:34:00 GMT
Repository: metron
Updated Branches:
  refs/heads/master 3fc8c84b9 -> eddbda88e


METRON-1220 Create documentation around alert nested field (justinleet) closes apache/metron#780


Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/eddbda88
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/eddbda88
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/eddbda88

Branch: refs/heads/master
Commit: eddbda88e4872479f1a78c3000cc6cf56f3567a6
Parents: 3fc8c84
Author: justinleet <justinjleet@gmail.com>
Authored: Fri Oct 6 09:33:23 2017 -0400
Committer: leet <leet@apache.org>
Committed: Fri Oct 6 09:33:23 2017 -0400

----------------------------------------------------------------------
 README.md                                      |  6 +++
 Upgrading.md                                   | 41 ++++++++++++++++
 metron-platform/metron-elasticsearch/README.md | 54 +++++++++++++++++++++
 metron-platform/metron-indexing/README.md      |  4 ++
 metron-platform/metron-parsers/README.md       |  6 +++
 5 files changed, 111 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 8599f0f..f3b2765 100644
--- a/README.md
+++ b/README.md
@@ -118,3 +118,9 @@ Some useful utilities that cross all of these parts of the architecture:
 * [Model as a Service](metron-analytics/metron-maas-service) : A Yarn application which can
deploy machine learning and statistical models onto the cluster along with the associated
Stellar functions to be able to call out to them in a scalable manner.
 * [Data management](metron-platform/metron-data-management) : A set of data management utilities
aimed at getting data into HBase in a format which will allow data flowing through metron
to be enriched with the results.  Contains integrations with threat intelligence feeds exposed
via TAXII as well as simple flat file structures.
 * [Profiler](metron-analytics/metron-profiler) : A feature extraction mechanism that can
generate a profile describing the behavior of an entity. An entity might be a server, user,
subnet or application. Once a profile has been generated defining what normal behavior looks-like,
models can be built that identify anomalous behavior.
+
+# Notes on Adding a New Sensor
+In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch
2.x,
+it is necessary to add an additional field to the templates and mapping for existing sensors.
+
+Please see a description of the steps necessary to make this change in the metron-elasticsearch
[Using Metron with Elasticsearch 2.x](./metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x)
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/Upgrading.md
----------------------------------------------------------------------
diff --git a/Upgrading.md b/Upgrading.md
index 6fb2486..dd68de0 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -2,6 +2,47 @@
 This document constitutes a per-version listing of changes of
 configuration which are non-backwards compatible.
 
+## 0.4.1 to 0.4.2
+
+### [METRON-1158: Build backend for grouping alerts into meta alerts](https://issues.apache.org/jira/browse/METRON-1158)
+In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch
2.x,
+it is necessary to add an additional field to the templates and mapping for existing sensors.
+
+Two steps must be done for each sensor, but not on each index for each sensor.
+
+First is to update the Elasticsearch template for each sensor, so any new indices have the
field:
+
+```
+export ELASTICSEARCH="node1"
+export SENSOR="bro"
+curl -XGET "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true" -o "${SENSOR}.template"
+sed -i '' '2d;$d' ./${SENSOR}.template
+sed -i '' '/"properties" : {/ a\
+"alert": { "type": "nested"},' ${SENSOR}.template
+curl -XPUT "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index" -d @${SENSOR}.template
+```
+
+To update existing indexes, update Elasticsearch mappings with the new field for each sensor.
 Make sure to set the ELASTICSEARCH variable appropriately.
+
+```
+curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc" -d '
+{
+        "properties" : {
+          "alert" : {
+            "type" : "nested"
+          }
+        }
+}
+'
+rm ${SENSOR}.template
+```
+
+For a more detailed description, please see metron-platform/metron-elasticsearch/README.md
+
+### Description
+
+In the 0.4.2 release, 
+
 ## 0.3.1 to 0.4.0
 
 ### [METRON-671: Refactor existing Ansible deployment to use Ambari MPack](https://issues.apache.org/jira/browse/METRON-671)

http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/metron-platform/metron-elasticsearch/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-elasticsearch/README.md b/metron-platform/metron-elasticsearch/README.md
new file mode 100644
index 0000000..2d2b139
--- /dev/null
+++ b/metron-platform/metron-elasticsearch/README.md
@@ -0,0 +1,54 @@
+# Elasticsearch in Metron
+
+## Introduction
+
+Elasticsearch can be used as the real-time portion of the datastore resulting from [metron-indexing](../metron-indexing.README.md).
+
+## Using Metron with Elasticsearch 2.x
+
+With Elasticsearch 2.x, there is a requirement that all sensors templates have a nested alert
field defined.  This field is a dummy field, and will be obsolete in Elasticsearch 5.x.  See
[Ignoring Unmapped Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields)
for more information
+
+Without this field, an error will be thrown during ALL searches (including from UIs, resulting
in no alerts being found for any sensor). This error will be found in the REST service's logs.
+
+Exception seen:
+```
+QueryParsingException[[nested] failed to find nested object under path [alert]];
+```
+
+There are two steps to resolve this issue.  First is to update the Elasticsearch template
for each sensor, so any new indices have the field. This requires retrieving the template,
removing an extraneous JSON field so we can put it back later, and adding our new field.
+
+Make sure to set the ELASTICSEARCH variable appropriately. $SENSOR can contain wildcards,
so if rollover has occurred, it's not necessary to do each index individually. The example
here appends `index*` to get all indexes for a the provided sensor.
+
+```
+export ELASTICSEARCH="node1"
+export SENSOR="bro"
+curl -XGET "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true" -o "${SENSOR}.template"
+sed -i '' '2d;$d' ./${SENSOR}.template
+sed -i '' '/"properties" : {/ a\
+"alert": { "type": "nested"},' ${SENSOR}.template
+```
+
+To manually verify this, you can optionally pretty print it again with:
+```
+python -m json.tool bro.template
+```
+
+We'll want to put the template back into Elasticsearch:
+```
+curl -XPUT "http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index" -d @${SENSOR}.template
+```
+
+To update existing indexes, update Elasticsearch mappings with the new field for each sensor.

+
+```
+curl -XPUT "http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc" -d '
+{
+        "properties" : {
+          "alert" : {
+            "type" : "nested"
+          }
+        }
+}
+'
+rm ${SENSOR}.template
+```

http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/metron-platform/metron-indexing/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md
index e65152c..6f47507 100644
--- a/metron-platform/metron-indexing/README.md
+++ b/metron-platform/metron-indexing/README.md
@@ -46,6 +46,10 @@ If unspecified, or set to `0`, it defaults to a system-determined duration
which
 parameter `topology.message.timeout.secs`.  Ignored if batchSize is `1`, since this disables
batching.
 * `enabled` : Whether the writer is enabled (default `true`).
 
+
+### Elasticsearch
+Metron comes with built-in templates for the default sensors for Elasticsearch. When adding
a new sensor, it will be necessary to add a new template defining the output fields appropriately.
In addition, there is a requirement for a field `alert` of type `nested` for Elasticsearch
2.x installs.  This is detailed at [Using Metron with Elasticsearch 2.x](../metron-elasticsearch/README.md#using-metron-with-elasticsearch-2x)
+
 ### Indexing Configuration Examples
 For a given  sensor, the following scenarios would be indicated by
 the following cases:

http://git-wip-us.apache.org/repos/asf/metron/blob/eddbda88/metron-platform/metron-parsers/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md
index 141e232..01eae71 100644
--- a/metron-platform/metron-parsers/README.md
+++ b/metron-platform/metron-parsers/README.md
@@ -434,6 +434,12 @@ and pass `--extra_topology_options custom_config.json` to `start_parser_topology
 Default installed Metron is untuned for production deployment.  There
 are a few knobs to tune to get the most out of your system.
 
+# Notes on Adding a New Sensor
+In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch
2.x,
+it is necessary to add an additional field to the templates and mapping for existing sensors.
+
+Please see a description of the steps necessary to make this change in the metron-elasticsearch
[Using Metron with Elasticsearch 2.x](./metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x)
+
 ## Kafka Queue
 The kafka queue associated with your parser is a collection point for
 all of the data sent to your parser.  As such, make sure that the number of partitions in


Mime
View raw message