metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mmiklav...@apache.org
Subject incubator-metron git commit: METRON-607: Enrichment doc improvement and test cleanup (mmiklavc) closes apache/incubator-metron#386
Date Mon, 05 Dec 2016 20:38:36 GMT
Repository: incubator-metron
Updated Branches:
  refs/heads/master 670f50ae3 -> e46539a98


METRON-607: Enrichment doc improvement and test cleanup (mmiklavc) closes apache/incubator-metron#386


Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/e46539a9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/e46539a9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/e46539a9

Branch: refs/heads/master
Commit: e46539a9802930e99eafbba7c74680e0fb48eaa3
Parents: 670f50a
Author: mmiklavc <michael.miklavcic@gmail.com>
Authored: Mon Dec 5 13:38:18 2016 -0700
Committer: Michael Miklavcic <michael.miklavcic@gmail.com>
Committed: Mon Dec 5 13:38:18 2016 -0700

----------------------------------------------------------------------
 metron-platform/metron-enrichment/README.md     | 14 ++++----
 .../integration/EnrichmentIntegrationTest.java  | 38 +++++++-------------
 2 files changed, 21 insertions(+), 31 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/e46539a9/metron-platform/metron-enrichment/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/README.md b/metron-platform/metron-enrichment/README.md
index 9527c8e..9f96f3e 100644
--- a/metron-platform/metron-enrichment/README.md
+++ b/metron-platform/metron-enrichment/README.md
@@ -48,8 +48,8 @@ The configuration is a complex JSON object with the following top level
fields:
 
 | Field            | Description                                                        
                                                                                         
                                                                | Example                
                                         |
 |------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------|
-| `fieldToTypeMap` | In the case of a simple HBase enrichment (i.e. a key/value lookup),
the mapping between fields and the enrichment types associated with those fields must be known.
 This enrichment type is used as part of the HBase key. | `"fieldToTypeMap" : { "ip_src_addr"
: [ "asset_enrichment" ] }`  |
-| `fieldMap`       | The map of enrichment bolts names to configuration handlers which know
how to split the message up.  The simplest of which is just a list of fields.  More complex
examples would be the stellar enrichment which provides stellar statements.  Each field is
sent to the enrichment referenced in the key.                                            
                                                    | `"fieldMap": {"hbaseEnrichment": ["ip_src_addr","ip_dst_addr"]}`
|
+| `fieldToTypeMap` | In the case of a simple HBase enrichment (i.e. a key/value lookup),
the mapping between fields and the enrichment types associated with those fields must be known.
 This enrichment type is used as part of the HBase key. Note: applies to hbaseEnrichment only.
| `"fieldToTypeMap" : { "ip_src_addr" : [ "asset_enrichment" ] }`  |
+| `fieldMap`       | The map of enrichment bolts names to configuration handlers which know
how to split the message up.  The simplest of which is just a list of fields.  More complex
examples would be the stellar enrichment which provides stellar statements. Each field listed
in the array arg is sent to the enrichment referenced in the key. Cardinality of fields to
enrichments is many-to-many. | `"fieldMap": {"hbaseEnrichment": ["ip_src_addr","ip_dst_addr"]}`
|
 | `config`         | The general configuration for the enrichment                       
                                                                                         
                                                                | `"config": {"typeToColumnFamily":
{ "asset_enrichment" : "cf" } }` |
 
 The `config` map is intended to house enrichment specific configuration.
@@ -73,7 +73,7 @@ The `fieldMap`contents are of interest because they contain the routing
and conf
       ]
       }
 ```
-For the `geo`, `host` and `hbaseEnrichment`, this is sufficient.  However, more complex enrichments
may contain their own configuration.  Currently, the `stellar` enrichment requires a more
complex configuration, such as:
+Based on this sample config, both ip_src_addr and ip_dst_addr will go to the `geo`, `host`,
and `hbaseEnrichment` adapter bolts. For the `geo`, `host` and `hbaseEnrichment`, this is
sufficient.  However, more complex enrichments may contain their own configuration.  Currently,
the `stellar` enrichment requires a more complex configuration, such as:
 ```
     "fieldMap": {
        ...
@@ -106,14 +106,14 @@ The other way in which the stellar enrichment is somewhat more complex
is in how
       }
     }
 ```
-We have a group called `numeric` whose stellar statements will be executed sequentially.
 In parallel to that, we have the group of stellar statements under the group `text` executing.
 The intent here is to allow you to not force higher latency operations to be done sequentially.
+We have a group called `numeric` whose stellar statements will be executed sequentially.
 In parallel to that, we have the group of stellar statements under the group `text` executing.
 The intent here is to allow you to not force higher latency operations to be done sequentially.
You can use any name for your groupings you like. Be aware that the configuration is a map
and duplicate configuration keys' values are not combined, so the duplicate configuration
value will be overwritten.
 
 ###The `threatIntel` Configuration 
 
 | Field            | Description                                                        
                                                                                         
                                                                                | Example
                                                                 |
 |------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|
-| `fieldToTypeMap` | In the case of a simple HBase threat intel enrichment (i.e. a key/value
lookup), the mapping between fields and the enrichment types associated with those fields
must be known.  This enrichment type is used as part of the HBase key. | `"fieldToTypeMap"
: { "ip_src_addr" : [ "malicious_ips" ] }`             |
-| `fieldMap`       | The map of threat intel enrichment bolts names to fields in the JSON
messages. Each field is sent to the threat intel enrichment bolt referenced in the key.  
                                                                           | `"fieldMap":
{"hbaseThreatIntel": ["ip_src_addr","ip_dst_addr"]}`        |
+| `fieldToTypeMap` | In the case of a simple HBase threat intel enrichment (i.e. a key/value
lookup), the mapping between fields and the enrichment types associated with those fields
must be known.  This enrichment type is used as part of the HBase key. Note: applies to hbaseThreatIntel
only. | `"fieldToTypeMap" : { "ip_src_addr" : [ "malicious_ips" ] }`             |
+| `fieldMap`       | The map of threat intel enrichment bolts names to fields in the JSON
messages. Each field is sent to the threat intel enrichment bolt referenced in the key. Each
field listed in the array arg is sent to the enrichment referenced in the key. Cardinality
of fields to enrichments is many-to-many.                                                
    | `"fieldMap": {"hbaseThreatIntel": ["ip_src_addr","ip_dst_addr"]}`        |
 | `triageConfig`   | The configuration of the threat triage scorer.  In the situation where
a threat is detected, a score is assigned to the message and embedded in the indexed message.
                                                                   | `"riskLevelRules" : {
"IN_SUBNET(ip_dst_addr, '192.168.0.0/24')" : 10 }` |
 | `config`         | The general configuration for the Threat Intel                     
                                                                                         
                                                                                | `"config":
{"typeToColumnFamily": { "malicious_ips","cf" } }`            |
 
@@ -190,6 +190,8 @@ An example configuration for the YAF sensor is as follows:
 }
 ```
 
+ThreatIntel alert levels are emitted as a new field "threat.triage.level." So for the example
above, an incoming message that trips the `ip_src_addr` rule will have a new field threat.triage.level=10.
+
 # Example Enrichment via Stellar
 
 Let's walk through doing a simple enrichment using Stellar on your cluster using the Squid
topology.

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/e46539a9/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
----------------------------------------------------------------------
diff --git a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
index a0199a1..d9c1dec 100644
--- a/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
+++ b/metron-platform/metron-enrichment/src/test/java/org/apache/metron/enrichment/integration/EnrichmentIntegrationTest.java
@@ -18,33 +18,32 @@
 package org.apache.metron.enrichment.integration;
 
 import com.fasterxml.jackson.core.type.TypeReference;
-import com.google.common.base.*;
-
+import com.google.common.base.Joiner;
+import com.google.common.base.Predicate;
+import com.google.common.base.Predicates;
+import com.google.common.base.Splitter;
 import com.google.common.collect.Iterables;
 import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.hbase.client.HTableInterface;
 import org.apache.metron.TestConstants;
 import org.apache.metron.common.Constants;
-import org.apache.metron.common.configuration.EnrichmentConfigurations;
+import org.apache.metron.common.utils.JSONUtils;
 import org.apache.metron.enrichment.bolt.ErrorEnrichmentBolt;
+import org.apache.metron.enrichment.converter.EnrichmentHelper;
+import org.apache.metron.enrichment.converter.EnrichmentKey;
+import org.apache.metron.enrichment.converter.EnrichmentValue;
+import org.apache.metron.enrichment.integration.components.ConfigUploadComponent;
+import org.apache.metron.enrichment.integration.mock.MockGeoAdapter;
+import org.apache.metron.enrichment.lookup.LookupKV;
 import org.apache.metron.enrichment.lookup.accesstracker.PersistentBloomTrackerCreator;
 import org.apache.metron.enrichment.stellar.SimpleHBaseEnrichmentFunctions;
 import org.apache.metron.hbase.TableProvider;
-import org.apache.metron.enrichment.converter.EnrichmentKey;
-import org.apache.metron.enrichment.converter.EnrichmentValue;
-import org.apache.metron.enrichment.converter.EnrichmentHelper;
 import org.apache.metron.integration.*;
-import org.apache.metron.enrichment.integration.components.ConfigUploadComponent;
+import org.apache.metron.integration.components.FluxTopologyComponent;
 import org.apache.metron.integration.components.KafkaComponent;
 import org.apache.metron.integration.components.ZKServerComponent;
 import org.apache.metron.integration.utils.TestUtils;
-import org.apache.metron.integration.components.FluxTopologyComponent;
-import org.apache.metron.enrichment.integration.mock.MockGeoAdapter;
 import org.apache.metron.test.mock.MockHTable;
-import org.apache.metron.enrichment.lookup.LookupKV;
-
-import org.apache.metron.enrichment.integration.utils.SampleUtil;
-import org.apache.metron.common.utils.JSONUtils;
 import org.junit.Assert;
 import org.junit.Test;
 
@@ -52,13 +51,7 @@ import javax.annotation.Nullable;
 import java.io.File;
 import java.io.IOException;
 import java.io.Serializable;
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
+import java.util.*;
 
 public class EnrichmentIntegrationTest extends BaseIntegrationTest {
   private static final String SRC_IP = "ip_src_addr";
@@ -68,11 +61,8 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
   private static final Map<String, Object> PLAYFUL_ENRICHMENT = new HashMap<String,
Object>() {{
     put("orientation", "north");
   }};
-  protected String testSensorType = "test";
-  protected String hdfsDir = "target/enrichmentIntegrationTest/hdfs";
   protected String fluxPath = "../metron-enrichment/src/main/flux/enrichment/test.yaml";
   protected String sampleParsedPath = TestConstants.SAMPLE_DATA_PARSED_PATH + "TestExampleParsed";
-  private String sampleIndexedPath = TestConstants.SAMPLE_DATA_INDEXED_PATH + "TestIndexed";
 
   public static class Provider implements TableProvider, Serializable {
     MockHTable.Provider  provider = new MockHTable.Provider();
@@ -84,8 +74,6 @@ public class EnrichmentIntegrationTest extends BaseIntegrationTest {
 
   @Test
   public void test() throws Exception {
-    final EnrichmentConfigurations configurations = SampleUtil.getSampleEnrichmentConfigs();
-    final String dateFormat = "yyyy.MM.dd.HH";
     final List<byte[]> inputMessages = TestUtils.readSampleData(sampleParsedPath);
     final String cf = "cf";
     final String trackerHBaseTableName = "tracker";


Mime
View raw message