metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ceste...@apache.org
Subject metron git commit: METRON-1235: Document the properties pulled from the global configuration closes apache/incubator-metron#791
Date Mon, 09 Oct 2017 13:39:15 GMT
Repository: metron
Updated Branches:
  refs/heads/master 830e5ccf4 -> b47c9a572


METRON-1235: Document the properties pulled from the global configuration closes apache/incubator-metron#791


Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/b47c9a57
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/b47c9a57
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/b47c9a57

Branch: refs/heads/master
Commit: b47c9a57260faa006173980a765cd565af7f12dd
Parents: 830e5cc
Author: cstella <cestella@gmail.com>
Authored: Mon Oct 9 09:39:04 2017 -0400
Committer: cstella <cestella@gmail.com>
Committed: Mon Oct 9 09:39:04 2017 -0400

----------------------------------------------------------------------
 metron-platform/metron-common/README.md        | 27 ++++++++++++++++++-
 metron-platform/metron-elasticsearch/README.md | 29 +++++++++++++++++++++
 metron-platform/metron-enrichment/README.md    | 21 ++++++++++++++-
 metron-platform/metron-indexing/README.md      | 13 +++++++++
 metron-platform/metron-parsers/README.md       | 17 +++++++++++-
 5 files changed, 104 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index 54738f8..5f9fec6 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -47,7 +47,7 @@ This configuration is stored in zookeeper, but looks something like
   "es.ip": "node1",
   "es.port": "9300",
   "es.date.format": "yyyy.MM.dd.HH",
-  "parser.error.topic": "indexing"
+  "parser.error.topic": "indexing",
   "fieldValidations" : [
               {
                 "input" : [ "ip_src_addr", "ip_dst_addr" ],
@@ -60,6 +60,31 @@ This configuration is stored in zookeeper, but looks something like
 }
 ```
 
+Various parts of our stack uses the global config are documented throughout the Metron documentation,
+but a convenient index is provided here:
+
+| Property Name                                                                         
                             | Subsystem     | Type       | Ambari Property            |
+|---------------------------------------------------------------------------------------------------------------------|---------------|------------|----------------------------|
+| [`es.clustername`](../metron-elasticsearch#esclustername)                             
                             | Indexing      | String     | `es_cluster_name`          |
+| [`es.ip`](../metron-elasticsearch#esip)                                               
                             | Indexing      | String     | `es_hosts`                 |
+| [`es.port`](../metron-elasticsearch#esport)                                           
                             | Indexing      | String     | `es_port`                  |
+| [`es.date.format`](../metron-elasticsearch#esdateformat)                              
                             | Indexing      | String     | `es_date_format`           |
+| [`fieldValidations`](#validation-framework)                                           
                             | Parsing       | Object     | N/A                        |
+| [`parser.error.topic`](../metron-parsers#parsererrortopic)                            
                             | Parsing       | String     | N/A                        |
+| [`stellar.function.paths`](../../metron-stellar/stellar-common#stellarfunctionpaths)  
                             | Stellar       | CSV String | N/A                        |
+| [`stellar.function.resolver.includes`](../../metron-stellar/stellar-common#stellarfunctionresolverincludesexcludes)
| Stellar       | CSV String | N/A                        |
+| [`stellar.function.resolver.excludes`](../../metron-stellar/stellar-common#stellarfunctionresolverincludesexcludes)
| Stellar       | CSV String | N/A                        |
+| [`profiler.period.duration`](../../metron-analytics/metron-profiler#profilerperiodduration)
                        | Profiler      | Integer    | `profiler_period_duration` |
+| [`profiler.period.duration.units`](../../metron-analytics/metron-profiler#profilerperioddurationunits)
             | Profiler      | String     | `profiler_period_units`    |
+| [`update.hbase.table`](../metron-indexing#updatehbasetable)                           
                             | REST/Indexing | String     | `update_hbase_table`       |
+| [`update.hbase.cf`](../metron-indexing#updatehbasecf)                                 
                             | REST/Indexing | String     | `update_hbase_cf`          |
+| [`geo.hdfs.file`](../metron-enrichment#geohdfsfile)                                   
                             | Enrichment    | String     | `geo_hdfs_file`            |
+
+## Note Configs in Ambari
+If a field is managed via ambari, you should change the field via
+ambari.  Otherwise, upon service restarts, you may find your update
+overwritten.
+
 # Validation Framework
 
 Inside of the global configuration, there is a validation framework in

http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-elasticsearch/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-elasticsearch/README.md b/metron-platform/metron-elasticsearch/README.md
index 2d2b139..1e39691 100644
--- a/metron-platform/metron-elasticsearch/README.md
+++ b/metron-platform/metron-elasticsearch/README.md
@@ -4,6 +4,35 @@
 
 Elasticsearch can be used as the real-time portion of the datastore resulting from [metron-indexing](../metron-indexing.README.md).
 
+## Properties
+
+### `es.clustername`
+
+The name of the elasticsearch Cluster.  See [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#cluster.name)
+
+### `es.ip`
+
+Specifies the nodes in the elasticsearch cluster to use for writing.
+The format is one of the following:
+* A hostname or IP address with a port (e.g. `hostname1:1234`), in which case `es.port` is
ignored.
+* A hostname or IP address without a port (e.g. `hostname1`), in which case `es.port` is
used.
+* A string containing a CSV of hostnames without ports (e.g. `hostname1,hostname2,hostname3`)
without spaces between.  `es.port` is assumed to be the port for each host.
+* A string containing a CSV of hostnames with ports (e.g. `hostname1:1234,hostname2:1234,hostname3:1234`)
without spaces between.  `es.port` is ignored.
+* A list of hostnames with ports (e.g. `[ "hostname1:1234", "hostname2:1234"]`).  Note, `es.port`
is NOT used in this construction.
+
+### `es.port`
+
+The port for the elasticsearch hosts.  This will be used in accordance with the discussion
of `es.ip`.
+
+### `es.date.format`
+
+The date format to use when constructing the indices.  For every message, the date format
will be applied
+to the current time and that will become the last part of the index name where the message
is written to.
+
+For instance, an `es.date.format` of `yyyy.MM.dd.HH` would have the consequence that the
indices would
+roll hourly, whereas an `es.date.format` of `yyyy.MM.dd` would have the consequence that
the indices would
+roll daily.
+
 ## Using Metron with Elasticsearch 2.x
 
 With Elasticsearch 2.x, there is a requirement that all sensors templates have a nested alert
field defined.  This field is a dummy field, and will be obsolete in Elasticsearch 5.x.  See
[Ignoring Unmapped Fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields)
for more information

http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-enrichment/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/README.md b/metron-platform/metron-enrichment/README.md
index 10f2cd4..70bf832 100644
--- a/metron-platform/metron-enrichment/README.md
+++ b/metron-platform/metron-enrichment/README.md
@@ -25,9 +25,27 @@ defined by JSON documents stored in zookeeper.
 There are two types of configurations at the moment, `global` and
 `sensor` specific.  
 
+
 ## Global Configuration 
 
-See the "[Global Configuration](../metron-common)" section.
+There are a few enrichments which have independent configurations, such
+as from the global config.
+
+Also, see the "[Global Configuration](../metron-common)" section for
+more discussion of the global config.
+
+### GeoIP
+Metron supports enrichment of IP information using
+[GeoLite2](https://dev.maxmind.com/geoip/geoip2/geolite2/). The
+location of the file is managed in the global config.
+
+#### `geo.hdfs.file`
+
+The location on HDFS of the GeoLite2 database file to use for GeoIP
+lookups.  This file will be localized on the storm supervisors running
+the topology and used from there. This is lazy, so if this property
+changes in a running topology, the file will be localized from HDFS upon first
+time the file is used via the geo enrichment. 
 
 ## Sensor Enrichment Configuration
 
@@ -269,6 +287,7 @@ An example configuration for the YAF sensor is as follows:
 
 ThreatIntel alert levels are emitted as a new field "threat.triage.level." So for the example
above, an incoming message that trips the `ip_src_addr` rule will have a new field threat.triage.level=10.
 
+
 # Example Enrichment via Stellar
 
 Let's walk through doing a simple enrichment using Stellar on your cluster using the Squid
topology.

http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-indexing/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-indexing/README.md b/metron-platform/metron-indexing/README.md
index 6f47507..6dbcb98 100644
--- a/metron-platform/metron-indexing/README.md
+++ b/metron-platform/metron-indexing/README.md
@@ -150,6 +150,19 @@ in parallel.  This enables a flexible strategy for specifying your backing
store
 For instance, currently the REST API supports the update functionality and may be configured
with a list of
 IndexDao implementations to use to support the updates.
 
+### The `HBaseDao`
+
+Updates will be written to HBase. The key structure is the GUID and
+for each new version, a new column is created with value as the message.
+
+The HBase table and column family are configured via fields in the global configuration.
+
+#### `update.hbase.table`
+The HBase table to use for message updates.
+
+#### `update.hbase.cf`
+The HBase column family to use for message updates.
+
 ### The `MetaAlertDao`
 
 The goal of meta alerts is to be able to group together a set of alerts while being able
to transparently perform actions

http://git-wip-us.apache.org/repos/asf/metron/blob/b47c9a57/metron-platform/metron-parsers/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-parsers/README.md b/metron-platform/metron-parsers/README.md
index 01eae71..75559c0 100644
--- a/metron-platform/metron-parsers/README.md
+++ b/metron-platform/metron-parsers/README.md
@@ -76,7 +76,22 @@ So putting it all together a typical Metron message with all 5-tuple fields
pres
 
 ## Global Configuration 
 
-See the "[Global Configuration](../metron-common)" section.
+There are a few properties which can be managed in the global configuration that have pertinence
to
+parsers and parsing in general.
+
+### `parser.error.topic`
+
+The topic where messages which were unable to be parsed due to error are sent.
+Error messages will be indexed under a sensor type of `error` and the messages will have
+the following fields:
+* `sensor.type`: `error`
+* `failed_sensor_type` : The sensor type of the message which wasn't able to be parsed
+* `error_type` : The error type, in this case `parser`.
+* `stack` : The stack trace of the error
+* `hostname` : The hostname of the node where the error happened
+* `raw_message` : The raw message in string form
+* `raw_message_bytes` : The raw message bytes
+* `error_hash` : A hash of the error message
 
 ## Parser Configuration
 


Mime
View raw message