metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ceste...@apache.org
Subject svn commit: r18345 - /dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-platform/metron-data-management/index.html
Date Wed, 15 Feb 2017 15:02:57 GMT
Author: cestella
Date: Wed Feb 15 15:02:57 2017
New Revision: 18345

Log:
Updating artifacts for Metron 0.3.1-RC4 (incubating)

Modified:
    dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-platform/metron-data-management/index.html

Modified: dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-platform/metron-data-management/index.html
==============================================================================
--- dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-platform/metron-data-management/index.html
(original)
+++ dev/incubator/metron/0.3.1-RC4-incubating/book-site/metron-platform/metron-data-management/index.html
Wed Feb 15 15:02:57 2017
@@ -1,13 +1,13 @@
 <!DOCTYPE html>
 <!--
- | Generated by Apache Maven Doxia at 2017-02-10
+ | Generated by Apache Maven Doxia at 2017-02-15
  | Rendered using Apache Maven Fluido Skin 1.3.0
 -->
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
-    <meta name="Date-Revision-yyyymmdd" content="20170210" />
+    <meta name="Date-Revision-yyyymmdd" content="20170215" />
     <meta http-equiv="Content-Language" content="en" />
     <title>Metron &#x2013; Resource Data Management</title>
     <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
@@ -64,7 +64,7 @@
         
                 
                     
-                  <li id="publishDate" class="pull-right">Last Published: 2017-02-10</li>
<li class="divider pull-right">|</li>
+                  <li id="publishDate" class="pull-right">Last Published: 2017-02-15</li>
<li class="divider pull-right">|</li>
               <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
             
                             </ul>
@@ -206,19 +206,86 @@
         <div id="bodyColumn"  class="span9" >
                                   
             <h1>Resource Data Management</h1>
+<p><a name="Resource_Data_Management"></a></p>
 <p>This project is a collection of classes to assist with loading of various enrichment
and threat intelligence sources into Metron.</p>
 <div class="section">
 <h2><a name="Simple_HBase_EnrichmentsThreat_Intelligence"></a>Simple HBase
Enrichments/Threat Intelligence</h2>
-<p>The vast majority of enrichments and threat intelligence processing tend toward
the following pattern: * Take a field * Look up the field in a key/value store * If the key
exists, then either it&#x2019;s a threat to be alerted or it should be enriched with the
value associated with the key.</p>
-<p>As such, we have created this capability as a default threat intel and enrichment
adapter. The basic primitive for simple enrichments and threat intelligence sources is a complex
key containing the following: * Type : The type of threat intel or enrichment (e.g. malicious_ip)
* Indicator : The indicator in question * Value : The value to associate with the type, indicator
pair. This is a JSON map.</p>
+<p>The vast majority of enrichments and threat intelligence processing tend toward
the following pattern:</p>
+
+<ul>
+  
+<li>Take a field</li>
+  
+<li>Look up the field in a key/value store</li>
+  
+<li>If the key exists, then either it&#x2019;s a threat to be alerted or it should
be enriched with the value associated with the key.</li>
+</ul>
+<p>As such, we have created this capability as a default threat intel and enrichment
adapter. The basic primitive for simple enrichments and threat intelligence sources is a complex
key containing the following:</p>
+
+<ul>
+  
+<li>Type : The type of threat intel or enrichment (e.g. malicious_ip)</li>
+  
+<li>Indicator : The indicator in question</li>
+  
+<li>Value : The value to associate with the type, indicator pair. This is a JSON map.</li>
+</ul>
 <p>At present, all of the dataloads utilities function by converting raw data sources
to this primitive key (type, indicator) and value to be placed in HBase.</p>
-<p>In the case of threat intel, a hit on the threat intel table will result in: * The
<tt>is_alert</tt> field being set to <tt>true</tt> in the index *
A field named <tt>threatintels.hbaseThreatIntel.$field.$threatintel_type</tt>
is set to <tt>alert</tt>  * <tt>$field</tt> is the field in the original
document that was a match (e.g. <tt>src_ip_addr</tt>)  * <tt>$threatintel_type</tt>
is the type of threat intel imported (defined in the Extractor configuration below).</p>
-<p>In the case of simple hbase enrichment, a hit on the enrichments table will result
in the following new field for each key in the value:<tt>enrichments.hbaseEnrichment.$field.$enrichment_type.$key</tt>
* <tt>$field</tt> is the field in the original document that was a match (e.g.
<tt>src_ip_addr</tt>) * <tt>$enrichment_type</tt> is the type of enrichment
imported (defined in the Extractor configuration below). * <tt>$key</tt> is a
key in the JSON map associated with the row in HBase.</p>
-<p>For instance, in the situation where we had the following very silly key/value in
HBase in the enrichment table: * indicator: <tt>127.0.0.1</tt> * type : <tt>important_addresses</tt>
* value: <tt>{ &quot;name&quot; : &quot;localhost&quot;, &quot;location&quot;
: &quot;home&quot; }</tt></p>
-<p>If we had a document whose <tt>ip_src_addr</tt> came through with a
value of <tt>127.0.0.1</tt>, we would have the following fields added to the indexed
document: * <tt>enrichments.hbaseEnrichment.ip_src_addr.important_addresses.name</tt>
: <tt>localhost</tt> * <tt>enrichments.hbaseEnrichment.ip_src_addr.important_addresses.location</tt>
: <tt>home</tt></p></div>
+<p>In the case of threat intel, a hit on the threat intel table will result in:</p>
+
+<ul>
+  
+<li>The <tt>is_alert</tt> field being set to <tt>true</tt>
in the index</li>
+  
+<li>A field named <tt>threatintels.hbaseThreatIntel.$field.$threatintel_type</tt>
is set to <tt>alert</tt>
+  
+<ul>
+    
+<li><tt>$field</tt> is the field in the original document that was a match
(e.g. <tt>src_ip_addr</tt>)</li>
+    
+<li><tt>$threatintel_type</tt> is the type of threat intel imported (defined
in the Extractor configuration below).</li>
+  </ul></li>
+</ul>
+<p>In the case of simple hbase enrichment, a hit on the enrichments table will result
in the following new field for each key in the value:<tt>enrichments.hbaseEnrichment.$field.$enrichment_type.$key</tt>
</p>
+
+<ul>
+  
+<li><tt>$field</tt> is the field in the original document that was a match
(e.g. <tt>src_ip_addr</tt>)</li>
+  
+<li><tt>$enrichment_type</tt> is the type of enrichment imported (defined
in the Extractor configuration below).</li>
+  
+<li><tt>$key</tt> is a key in the JSON map associated with the row in HBase.</li>
+</ul>
+<p>For instance, in the situation where we had the following very silly key/value in
HBase in the enrichment table:</p>
+
+<ul>
+  
+<li>indicator: <tt>127.0.0.1</tt></li>
+  
+<li>type : <tt>important_addresses</tt></li>
+  
+<li>value: <tt>{ &quot;name&quot; : &quot;localhost&quot;, &quot;location&quot;
: &quot;home&quot; }</tt></li>
+</ul>
+<p>If we had a document whose <tt>ip_src_addr</tt> came through with a
value of <tt>127.0.0.1</tt>, we would have the following fields added to the indexed
document:</p>
+
+<ul>
+  
+<li><tt>enrichments.hbaseEnrichment.ip_src_addr.important_addresses.name</tt>
: <tt>localhost</tt></li>
+  
+<li><tt>enrichments.hbaseEnrichment.ip_src_addr.important_addresses.location</tt>
: <tt>home</tt></li>
+</ul></div>
 <div class="section">
 <h2><a name="Extractor_Framework"></a>Extractor Framework</h2>
-<p>For the purpose of ingesting data of a variety of formats, we have created an Extractor
framework which allows for common data formats to be interpreted as enrichment or threat intelligence
sources. The formats supported at present are: * CSV (both threat intel and enrichment) *
STIX (threat intel only) * Custom (pass your own class)</p>
+<p>For the purpose of ingesting data of a variety of formats, we have created an Extractor
framework which allows for common data formats to be interpreted as enrichment or threat intelligence
sources. The formats supported at present are:</p>
+
+<ul>
+  
+<li>CSV (both threat intel and enrichment)</li>
+  
+<li>STIX (threat intel only)</li>
+  
+<li>Custom (pass your own class)</li>
+</ul>
 <p>All of the current utilities take a JSON file to configure how to interpret input
data. This JSON describes the type of data and the schema if necessary for the data if it
is not fixed (as in STIX, e.g.).</p>
 <div class="section">
 <h3><a name="CSV_Extractor"></a>CSV Extractor</h3>
@@ -240,7 +307,16 @@
 }
 </pre></div></div>
 <p>In this example, we have instructed the extractor of the schema (i.e. the columns
field), two columns at the first and third position. We have indicated that the <tt>ip</tt>
column is the indicator type and that the enrichment type is named <tt>malicious_ip</tt>.
We have also indicated that the extractor to use is the CSV Extractor. The other option is
the STIX extractor or a fully qualified classname for your own extractor.</p>
-<p>The meta column values will show up in the value in HBase because it is called out
as a non-indicator column. The key for the value will be &#x2018;meta&#x2019;. For
instance, given an input string of <tt>123.45.123.12,something,the grapevine</tt>,
the following key, value would be extracted: * Indicator : <tt>123.45.123.12</tt>
* Type : <tt>malicious_ip</tt> * Value : <tt>{ &quot;ip&quot; :
&quot;123.45.123.12&quot;, &quot;source&quot; : &quot;the grapevine&quot;
}</tt></p></div>
+<p>The meta column values will show up in the value in HBase because it is called out
as a non-indicator column. The key for the value will be &#x2018;meta&#x2019;. For
instance, given an input string of <tt>123.45.123.12,something,the grapevine</tt>,
the following key, value would be extracted:</p>
+
+<ul>
+  
+<li>Indicator : <tt>123.45.123.12</tt></li>
+  
+<li>Type : <tt>malicious_ip</tt></li>
+  
+<li>Value : <tt>{ &quot;ip&quot; : &quot;123.45.123.12&quot;,
&quot;source&quot; : &quot;the grapevine&quot; }</tt></li>
+</ul></div>
 <div class="section">
 <h3><a name="STIX_Extractor"></a>STIX Extractor</h3>
 <p>Consider the following config for importing STIX documents. This is a threat intelligence
interchange format, so it is particularly relevant and attractive data to import for our purposes.
Because STIX is a standard format, there is no need to specify the schema or how to interpret
the documents.</p>
@@ -400,11 +476,14 @@
     </tr>
   </tbody>
 </table>
-<p>top-list.csv <tt>
-1,google.com
+<p>top-list.csv</p>
+
+<div class="source">
+<div class="source">
+<pre>1,google.com
 2,youtube.com
 ...
-</tt></p>
+</pre></div></div>
 <p>Running a file import with the above data and extractor configuration would result
in the following 2 extracted data records:</p>
 
 <table border="0" class="table table-striped">
@@ -440,13 +519,26 @@
     </tr>
   </tbody>
 </table>
-<p>Similar to the parser framework, providing a Zookeeper quorum via the zk_quorum
property will enable Stellar to access properties that reside in the global config. Expanding
on our example above, if the global config looks as follows: <tt>
-{
+<p>Similar to the parser framework, providing a Zookeeper quorum via the zk_quorum
property will enable Stellar to access properties that reside in the global config. Expanding
on our example above, if the global config looks as follows:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
     &quot;global_property&quot; : &quot;metron-ftw&quot;
 }
-</tt></p>
-<p>And we expand our value_tranform: ```` &#x2026;  &#x201c;value_transform&#x201d;
: {  &#x201c;domain&#x201d; : &#x201c;DOMAIN_REMOVE_TLD(domain)&#x201d;, 
&#x201c;a-new-prop&#x201d; : &#x201c;global_property&#x201d;  }, &#x2026;</p>
-<p>````</p>
+</pre></div></div>
+<p>And we expand our value_tranform:</p>
+
+<div class="source">
+<div class="source">
+<pre>...
+    &quot;value_transform&quot; : {
+       &quot;domain&quot; : &quot;DOMAIN_REMOVE_TLD(domain)&quot;,
+       &quot;a-new-prop&quot; : &quot;global_property&quot;
+    },
+...
+
+</pre></div></div>
 <p>The resulting value data would look like the following:</p>
 
 <table border="0" class="table table-striped">
@@ -502,11 +594,27 @@
                         }
 }
 </pre></div></div>
-<p>We have to specify the following: * The zookeeper quorum which holds the cluster
configuration * The mapping between the fields in the enriched documents and the enrichment
types.</p>
+<p>We have to specify the following:</p>
+
+<ul>
+  
+<li>The zookeeper quorum which holds the cluster configuration</li>
+  
+<li>The mapping between the fields in the enriched documents and the enrichment types.</li>
+</ul>
 <p>This configuration allows the ingestion tools to update zookeeper post-ingestion
so that the enrichment topology can take advantage immediately of the new type.</p></div>
 <div class="section">
 <h2><a name="Loading_Utilities"></a>Loading Utilities</h2>
-<p>The two configurations above are used in the three separate ingestion tools: * Taxii
Loader * Bulk load from HDFS via MapReduce * Flat File ingestion</p>
+<p>The two configurations above are used in the three separate ingestion tools:</p>
+
+<ul>
+  
+<li>Taxii Loader</li>
+  
+<li>Bulk load from HDFS via MapReduce</li>
+  
+<li>Flat File ingestion</li>
+</ul>
 <div class="section">
 <h3><a name="Taxii_Loader"></a>Taxii Loader</h3>
 <p>The shell script <tt>$METRON_HOME/bin/threatintel_taxii_load.sh</tt>
can be used to poll a Taxii server for STIX documents and ingest them into HBase.<br />It
is quite common for this Taxii server to be an aggregation server such as Soltra Edge.</p>
@@ -523,7 +631,22 @@
   ,&quot;allowedIndicatorTypes&quot; : [ &quot;domainname:FQDN&quot;, &quot;address:IPV_4_ADDR&quot;
]
 }
 </pre></div></div>
-<p>As you can see, we are specifying the following information: * endpoint : The URL
of the endpoint * type : <tt>POLL</tt> or <tt>DISCOVER</tt> depending
on the endpoint. * collection : The Taxii collection to ingest * table : The HBase table to
import into * columnFamily : The column family to import into * allowedIndicatorTypes : an
array of acceptable threat intel types (see the &#x201c;Enrichment Type Name&#x201d;
column of the Stix table above for the possibilities).</p>
+<p>As you can see, we are specifying the following information:</p>
+
+<ul>
+  
+<li>endpoint : The URL of the endpoint</li>
+  
+<li>type : <tt>POLL</tt> or <tt>DISCOVER</tt> depending on
the endpoint.</li>
+  
+<li>collection : The Taxii collection to ingest</li>
+  
+<li>table : The HBase table to import into</li>
+  
+<li>columnFamily : The column family to import into</li>
+  
+<li>allowedIndicatorTypes : an array of acceptable threat intel types (see the &#x201c;Enrichment
Type Name&#x201d; column of the Stix table above for the possibilities).</li>
+</ul>
 <p>The parameters for the utility are as follows:</p>
 
 <table border="0" class="table table-striped">
@@ -623,7 +746,12 @@
 <div class="section">
 <h3><a name="Flatfile_Loader"></a>Flatfile Loader</h3>
 <p>The shell script <tt>$METRON_HOME/bin/flatfile_loader.sh</tt> will read
data from local disk, HDFS or URLs and load the enrichment or threat intel data into an HBase
table.<br />Note: This utility works for enrichment as well as threat intel due to the
underlying infrastructure being the same.</p>
-<p>One special thing to note here is that there is a special configuration parameter
to the Extractor config that is only considered during this loader: * inputFormat : This specifies
how to consider the data. The two implementations are <tt>BY_LINE</tt> and <tt>WHOLE_FILE</tt>.</p>
+<p>One special thing to note here is that there is a special configuration parameter
to the Extractor config that is only considered during this loader:</p>
+
+<ul>
+  
+<li>inputFormat : This specifies how to consider the data. The two implementations
are <tt>BY_LINE</tt> and <tt>WHOLE_FILE</tt>.</li>
+</ul>
 <p>The default is <tt>BY_LINE</tt>, which makes sense for a list of CSVs
where each line indicates a unit of information which can be imported. However, if you are
importing a set of STIX documents, then you want each document to be considered as input to
the Extractor.</p>
 <p>The parameters for the utility are as follows:</p>
 



Mime
View raw message