metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ceste...@apache.org
Subject svn commit: r23994 [17/24] - in /release/metron: 0.4.1/ 0.4.2/ 0.4.2/site-book/ 0.4.2/site-book/css/ 0.4.2/site-book/images/ 0.4.2/site-book/images/logos/ 0.4.2/site-book/images/profiles/ 0.4.2/site-book/img/ 0.4.2/site-book/js/ 0.4.2/site-book/metron-...
Date Wed, 03 Jan 2018 18:25:58 GMT
Added: release/metron/0.4.2/site-book/metron-platform/metron-elasticsearch/index.html
==============================================================================
--- release/metron/0.4.2/site-book/metron-platform/metron-elasticsearch/index.html (added)
+++ release/metron/0.4.2/site-book/metron-platform/metron-elasticsearch/index.html Wed Jan  3 18:25:57 2018
@@ -0,0 +1,380 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-12-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta name="Date-Revision-yyyymmdd" content="20171208" />
+    <meta http-equiv="Content-Language" content="en" />
+    <title>Metron &#x2013; Elasticsearch in Metron</title>
+    <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+    <link rel="stylesheet" href="../../css/site.css" />
+    <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+      
+    <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+                          
+        
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+          
+            </head>
+        <body class="topBarDisabled">
+          
+                
+                    
+    
+        <div class="container-fluid">
+          <div id="banner">
+        <div class="pull-left">
+                                    <a href="http://metron.apache.org/" id="bannerLeft">
+                                                                                                <img src="../../images/metron-logo.png"  alt="Apache Metron" width="148px" height="48px"/>
+                </a>
+                      </div>
+        <div class="pull-right">  </div>
+        <div class="clear"><hr/></div>
+      </div>
+
+      <div id="breadcrumbs">
+        <ul class="breadcrumb">
+                
+                    
+                              <li class="">
+                    <a href="http://www.apache.org" class="externalLink" title="Apache">
+        Apache</a>
+        </li>
+      <li class="divider ">/</li>
+            <li class="">
+                    <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+        Metron</a>
+        </li>
+      <li class="divider ">/</li>
+            <li class="">
+                    <a href="../../index.html" title="Documentation">
+        Documentation</a>
+        </li>
+      <li class="divider ">/</li>
+        <li class="">Elasticsearch in Metron</li>
+        
+                
+                    
+                  <li id="publishDate" class="pull-right">Last Published: 2017-12-08</li> <li class="divider pull-right">|</li>
+              <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+            
+                            </ul>
+      </div>
+
+            
+      <div class="row-fluid">
+        <div id="leftColumn" class="span3">
+          <div class="well sidebar-nav">
+                
+                    
+                <ul class="nav nav-list">
+                    <li class="nav-header">User Documentation</li>
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                                                                          
+      <li>
+    
+                          <a href="../../index.html" title="Metron">
+          <i class="icon-chevron-down"></i>
+        Metron</a>
+                    <ul class="nav nav-list">
+                      
+      <li>
+    
+                          <a href="../../Upgrading.html" title="Upgrading">
+          <i class="none"></i>
+        Upgrading</a>
+            </li>
+                                                                                                                                                      
+      <li>
+    
+                          <a href="../../metron-analytics/index.html" title="Analytics">
+          <i class="icon-chevron-right"></i>
+        Analytics</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+          <i class="none"></i>
+        Docker</a>
+            </li>
+                                                                                                                                                                                                                                                                                                                                                                                                            
+      <li>
+    
+                          <a href="../../metron-deployment/index.html" title="Deployment">
+          <i class="icon-chevron-right"></i>
+        Deployment</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+          <i class="none"></i>
+        Alerts</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-config/index.html" title="Config">
+          <i class="none"></i>
+        Config</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+          <i class="none"></i>
+        Rest</a>
+            </li>
+                                                                                                                                                                                                                                                                                              
+      <li>
+    
+                          <a href="../../metron-platform/index.html" title="Platform">
+          <i class="icon-chevron-down"></i>
+        Platform</a>
+                    <ul class="nav nav-list">
+                      
+      <li>
+    
+                          <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide">
+          <i class="none"></i>
+        Performance-tuning-guide</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-api/index.html" title="Api">
+          <i class="none"></i>
+        Api</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-common/index.html" title="Common">
+          <i class="none"></i>
+        Common</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-data-management/index.html" title="Data-management">
+          <i class="none"></i>
+        Data-management</a>
+            </li>
+                      
+      <li class="active">
+    
+            <a href="#"><i class="none"></i>Elasticsearch</a>
+          </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
+          <i class="none"></i>
+        Enrichment</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-indexing/index.html" title="Indexing">
+          <i class="none"></i>
+        Indexing</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-management/index.html" title="Management">
+          <i class="none"></i>
+        Management</a>
+            </li>
+                                                                        
+      <li>
+    
+                          <a href="../../metron-platform/metron-parsers/index.html" title="Parsers">
+          <i class="icon-chevron-right"></i>
+        Parsers</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+          <i class="none"></i>
+        Pcap-backend</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+          <i class="none"></i>
+        Writer</a>
+            </li>
+              </ul>
+        </li>
+                                                                                          
+      <li>
+    
+                          <a href="../../metron-sensors/index.html" title="Sensors">
+          <i class="icon-chevron-right"></i>
+        Sensors</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+          <i class="none"></i>
+        Stellar-3rd-party-example</a>
+            </li>
+                                                                        
+      <li>
+    
+                          <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+          <i class="icon-chevron-right"></i>
+        Stellar-common</a>
+                  </li>
+                                                                                          
+      <li>
+    
+                          <a href="../../use-cases/index.html" title="Use-cases">
+          <i class="icon-chevron-right"></i>
+        Use-cases</a>
+                  </li>
+              </ul>
+        </li>
+            </ul>
+                
+                    
+                
+          <hr class="divider" />
+
+           <div id="poweredBy">
+                            <div class="clear"></div>
+                            <div class="clear"></div>
+                            <div class="clear"></div>
+                             <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+        <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+      </a>
+                  </div>
+          </div>
+        </div>
+        
+                
+        <div id="bodyColumn"  class="span9" >
+                                  
+            <h1>Elasticsearch in Metron</h1>
+<p><a name="Elasticsearch_in_Metron"></a></p>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>Elasticsearch can be used as the real-time portion of the datastore resulting from <a href="../metron-indexing/index.html">metron-indexing</a>.</p></div>
+<div class="section">
+<h2><a name="Properties"></a>Properties</h2>
+<div class="section">
+<h3><a name="es.clustername"></a><tt>es.clustername</tt></h3>
+<p>The name of the elasticsearch Cluster. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#cluster.name">here</a></p></div>
+<div class="section">
+<h3><a name="es.ip"></a><tt>es.ip</tt></h3>
+<p>Specifies the nodes in the elasticsearch cluster to use for writing. The format is one of the following:</p>
+
+<ul>
+  
+<li>A hostname or IP address with a port (e.g. <tt>hostname1:1234</tt>), in which case <tt>es.port</tt> is ignored.</li>
+  
+<li>A hostname or IP address without a port (e.g. <tt>hostname1</tt>), in which case <tt>es.port</tt> is used.</li>
+  
+<li>A string containing a CSV of hostnames without ports (e.g. <tt>hostname1,hostname2,hostname3</tt>) without spaces between. <tt>es.port</tt> is assumed to be the port for each host.</li>
+  
+<li>A string containing a CSV of hostnames with ports (e.g. <tt>hostname1:1234,hostname2:1234,hostname3:1234</tt>) without spaces between. <tt>es.port</tt> is ignored.</li>
+  
+<li>A list of hostnames with ports (e.g. <tt>[ &quot;hostname1:1234&quot;, &quot;hostname2:1234&quot;]</tt>). Note, <tt>es.port</tt> is NOT used in this construction.</li>
+</ul></div>
+<div class="section">
+<h3><a name="es.port"></a><tt>es.port</tt></h3>
+<p>The port for the elasticsearch hosts. This will be used in accordance with the discussion of <tt>es.ip</tt>.</p></div>
+<div class="section">
+<h3><a name="es.date.format"></a><tt>es.date.format</tt></h3>
+<p>The date format to use when constructing the indices. For every message, the date format will be applied to the current time and that will become the last part of the index name where the message is written to.</p>
+<p>For instance, an <tt>es.date.format</tt> of <tt>yyyy.MM.dd.HH</tt> would have the consequence that the indices would roll hourly, whereas an <tt>es.date.format</tt> of <tt>yyyy.MM.dd</tt> would have the consequence that the indices would roll daily.</p></div></div>
+<div class="section">
+<h2><a name="Using_Metron_with_Elasticsearch_2.x"></a>Using Metron with Elasticsearch 2.x</h2>
+<p>With Elasticsearch 2.x, there is a requirement that all sensors templates have a nested alert field defined. This field is a dummy field, and will be obsolete in Elasticsearch 5.x. See <a class="externalLink" href="https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields">Ignoring Unmapped Fields</a> for more information</p>
+<p>Without this field, an error will be thrown during ALL searches (including from UIs, resulting in no alerts being found for any sensor). This error will be found in the REST service&#x2019;s logs.</p>
+<p>Exception seen:</p>
+
+<div class="source">
+<div class="source">
+<pre>QueryParsingException[[nested] failed to find nested object under path [alert]];
+</pre></div></div>
+<p>There are two steps to resolve this issue. First is to update the Elasticsearch template for each sensor, so any new indices have the field. This requires retrieving the template, removing an extraneous JSON field so we can put it back later, and adding our new field.</p>
+<p>Make sure to set the ELASTICSEARCH variable appropriately. $SENSOR can contain wildcards, so if rollover has occurred, it&#x2019;s not necessary to do each index individually. The example here appends <tt>index*</tt> to get all indexes for a the provided sensor.</p>
+
+<div class="source">
+<div class="source">
+<pre>export ELASTICSEARCH=&quot;node1&quot;
+export SENSOR=&quot;bro&quot;
+curl -XGET &quot;http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index*?pretty=true&quot; -o &quot;${SENSOR}.template&quot;
+sed -i '' '2d;$d' ./${SENSOR}.template
+sed -i '' '/&quot;properties&quot; : {/ a\
+&quot;alert&quot;: { &quot;type&quot;: &quot;nested&quot;},' ${SENSOR}.template
+</pre></div></div>
+<p>To manually verify this, you can optionally pretty print it again with:</p>
+
+<div class="source">
+<div class="source">
+<pre>python -m json.tool bro.template
+</pre></div></div>
+<p>We&#x2019;ll want to put the template back into Elasticsearch:</p>
+
+<div class="source">
+<div class="source">
+<pre>curl -XPUT &quot;http://${ELASTICSEARCH}:9200/_template/${SENSOR}_index&quot; -d @${SENSOR}.template
+</pre></div></div>
+<p>To update existing indexes, update Elasticsearch mappings with the new field for each sensor. </p>
+
+<div class="source">
+<div class="source">
+<pre>curl -XPUT &quot;http://${ELASTICSEARCH}:9200/${SENSOR}_index*/_mapping/${SENSOR}_doc&quot; -d '
+{
+        &quot;properties&quot; : {
+          &quot;alert&quot; : {
+            &quot;type&quot; : &quot;nested&quot;
+          }
+        }
+}
+'
+rm ${SENSOR}.template
+</pre></div></div></div>
+<div class="section">
+<h2><a name="Installing_Elasticsearch_Templates"></a>Installing Elasticsearch Templates</h2>
+<p>The stock set of Elasticsearch templates for bro, snort, yaf, error index and meta index are installed automatically during the first time install and startup of Metron Indexing service.</p>
+<p>It is possible that Elasticsearch service is not available when the Metron Indexing Service startup, in that case the Elasticsearch template will not be installed. </p>
+<p>For such a scenario, an Admin can have the template installed in two ways:</p>
+<p><i>Method 1</i> - Manually from the Ambari UI by following the flow: Ambari UI -&gt; Services -&gt; Metron -&gt; Service Actions -&gt; Elasticsearch Template Install</p>
+<p><i>Method 2</i> - Stop the Metron Indexing service, and start it again from Ambari UI. Note that the Metron Indexing service tracks if it has successfully installed the Elasticsearch templates, and will attempt to do so each time it is Started until successful.</p>
+
+<blockquote>
+<p>Note: If you have made any customization to your index templates, then installing Elasticsearch templates afresh will lead to overwriting your existing changes. Please exercise caution.</p>
+</blockquote></div>
+                  </div>
+            </div>
+          </div>
+
+    <hr/>
+
+    <footer>
+            <div class="container-fluid">
+              <div class="row span12">Copyright &copy;                    2017
+                        <a href="https://www.apache.org">The Apache Software Foundation</a>.
+            All Rights Reserved.      
+                    
+      </div>
+
+                          
+        
+                </div>
+    </footer>
+  </body>
+</html>

Added: release/metron/0.4.2/site-book/metron-platform/metron-enrichment/index.html
==============================================================================
--- release/metron/0.4.2/site-book/metron-platform/metron-enrichment/index.html (added)
+++ release/metron/0.4.2/site-book/metron-platform/metron-enrichment/index.html Wed Jan  3 18:25:57 2018
@@ -0,0 +1,765 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-12-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta name="Date-Revision-yyyymmdd" content="20171208" />
+    <meta http-equiv="Content-Language" content="en" />
+    <title>Metron &#x2013; Enrichment</title>
+    <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+    <link rel="stylesheet" href="../../css/site.css" />
+    <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+      
+    <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+                          
+        
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+          
+            </head>
+        <body class="topBarDisabled">
+          
+                
+                    
+    
+        <div class="container-fluid">
+          <div id="banner">
+        <div class="pull-left">
+                                    <a href="http://metron.apache.org/" id="bannerLeft">
+                                                                                                <img src="../../images/metron-logo.png"  alt="Apache Metron" width="148px" height="48px"/>
+                </a>
+                      </div>
+        <div class="pull-right">  </div>
+        <div class="clear"><hr/></div>
+      </div>
+
+      <div id="breadcrumbs">
+        <ul class="breadcrumb">
+                
+                    
+                              <li class="">
+                    <a href="http://www.apache.org" class="externalLink" title="Apache">
+        Apache</a>
+        </li>
+      <li class="divider ">/</li>
+            <li class="">
+                    <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+        Metron</a>
+        </li>
+      <li class="divider ">/</li>
+            <li class="">
+                    <a href="../../index.html" title="Documentation">
+        Documentation</a>
+        </li>
+      <li class="divider ">/</li>
+        <li class="">Enrichment</li>
+        
+                
+                    
+                  <li id="publishDate" class="pull-right">Last Published: 2017-12-08</li> <li class="divider pull-right">|</li>
+              <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+            
+                            </ul>
+      </div>
+
+            
+      <div class="row-fluid">
+        <div id="leftColumn" class="span3">
+          <div class="well sidebar-nav">
+                
+                    
+                <ul class="nav nav-list">
+                    <li class="nav-header">User Documentation</li>
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                                                                          
+      <li>
+    
+                          <a href="../../index.html" title="Metron">
+          <i class="icon-chevron-down"></i>
+        Metron</a>
+                    <ul class="nav nav-list">
+                      
+      <li>
+    
+                          <a href="../../Upgrading.html" title="Upgrading">
+          <i class="none"></i>
+        Upgrading</a>
+            </li>
+                                                                                                                                                      
+      <li>
+    
+                          <a href="../../metron-analytics/index.html" title="Analytics">
+          <i class="icon-chevron-right"></i>
+        Analytics</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+          <i class="none"></i>
+        Docker</a>
+            </li>
+                                                                                                                                                                                                                                                                                                                                                                                                            
+      <li>
+    
+                          <a href="../../metron-deployment/index.html" title="Deployment">
+          <i class="icon-chevron-right"></i>
+        Deployment</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+          <i class="none"></i>
+        Alerts</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-config/index.html" title="Config">
+          <i class="none"></i>
+        Config</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+          <i class="none"></i>
+        Rest</a>
+            </li>
+                                                                                                                                                                                                                                                                                              
+      <li>
+    
+                          <a href="../../metron-platform/index.html" title="Platform">
+          <i class="icon-chevron-down"></i>
+        Platform</a>
+                    <ul class="nav nav-list">
+                      
+      <li>
+    
+                          <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide">
+          <i class="none"></i>
+        Performance-tuning-guide</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-api/index.html" title="Api">
+          <i class="none"></i>
+        Api</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-common/index.html" title="Common">
+          <i class="none"></i>
+        Common</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-data-management/index.html" title="Data-management">
+          <i class="none"></i>
+        Data-management</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+          <i class="none"></i>
+        Elasticsearch</a>
+            </li>
+                      
+      <li class="active">
+    
+            <a href="#"><i class="none"></i>Enrichment</a>
+          </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-indexing/index.html" title="Indexing">
+          <i class="none"></i>
+        Indexing</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-management/index.html" title="Management">
+          <i class="none"></i>
+        Management</a>
+            </li>
+                                                                        
+      <li>
+    
+                          <a href="../../metron-platform/metron-parsers/index.html" title="Parsers">
+          <i class="icon-chevron-right"></i>
+        Parsers</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+          <i class="none"></i>
+        Pcap-backend</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+          <i class="none"></i>
+        Writer</a>
+            </li>
+              </ul>
+        </li>
+                                                                                          
+      <li>
+    
+                          <a href="../../metron-sensors/index.html" title="Sensors">
+          <i class="icon-chevron-right"></i>
+        Sensors</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+          <i class="none"></i>
+        Stellar-3rd-party-example</a>
+            </li>
+                                                                        
+      <li>
+    
+                          <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+          <i class="icon-chevron-right"></i>
+        Stellar-common</a>
+                  </li>
+                                                                                          
+      <li>
+    
+                          <a href="../../use-cases/index.html" title="Use-cases">
+          <i class="icon-chevron-right"></i>
+        Use-cases</a>
+                  </li>
+              </ul>
+        </li>
+            </ul>
+                
+                    
+                
+          <hr class="divider" />
+
+           <div id="poweredBy">
+                            <div class="clear"></div>
+                            <div class="clear"></div>
+                            <div class="clear"></div>
+                             <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+        <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+      </a>
+                  </div>
+          </div>
+        </div>
+        
+                
+        <div id="bodyColumn"  class="span9" >
+                                  
+            <h1>Enrichment</h1>
+<p><a name="Enrichment"></a></p>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>The <tt>enrichment</tt> topology is a topology dedicated to taking the data from the parsing topologies that have been normalized into the Metron data format (e.g. a JSON Map structure with <tt>original_message</tt> and <tt>timestamp</tt>) and </p>
+
+<ul>
+  
+<li>Enriching messages with external data from data stores (e.g. hbase) by  adding new fields based on existing fields in the messages.</li>
+  
+<li>Marking messages as threats based on data in external data stores</li>
+  
+<li>Marking threat alerts with a numeric triage level based on a set of  Stellar rules.</li>
+</ul></div>
+<div class="section">
+<h2><a name="Enrichment_Architecture"></a>Enrichment Architecture</h2>
+<p><img src="../../images/enrichment_arch.png" alt="Architecture" /></p></div>
+<div class="section">
+<h2><a name="Enrichment_Configuration"></a>Enrichment Configuration</h2>
+<p>The configuration for the <tt>enrichment</tt> topology, the topology primarily responsible for enrichment and threat intelligence enrichment, is defined by JSON documents stored in zookeeper.</p>
+<p>There are two types of configurations at the moment, <tt>global</tt> and <tt>sensor</tt> specific. </p></div>
+<div class="section">
+<h2><a name="Global_Configuration"></a>Global Configuration</h2>
+<p>There are a few enrichments which have independent configurations, such as from the global config.</p>
+<p>Also, see the &#x201c;<a href="../metron-common/index.html">Global Configuration</a>&#x201d; section for more discussion of the global config.</p>
+<div class="section">
+<h3><a name="GeoIP"></a>GeoIP</h3>
+<p>Metron supports enrichment of IP information using <a class="externalLink" href="https://dev.maxmind.com/geoip/geoip2/geolite2/">GeoLite2</a>. The location of the file is managed in the global config.</p>
+<div class="section">
+<h4><a name="geo.hdfs.file"></a><tt>geo.hdfs.file</tt></h4>
+<p>The location on HDFS of the GeoLite2 database file to use for GeoIP lookups. This file will be localized on the storm supervisors running the topology and used from there. This is lazy, so if this property changes in a running topology, the file will be localized from HDFS upon first time the file is used via the geo enrichment. </p></div></div></div>
+<div class="section">
+<h2><a name="Sensor_Enrichment_Configuration"></a>Sensor Enrichment Configuration</h2>
+<p>The sensor specific configuration is intended to configure the individual enrichments and threat intelligence enrichments for a given sensor type (e.g. <tt>snort</tt>).</p>
+<p>Just like the global config, the format is a JSON stored in zookeeper. The configuration is a complex JSON object with the following top level fields:</p>
+
+<ul>
+  
+<li><tt>enrichment</tt> : A complex JSON object representing the configuration of the enrichments</li>
+  
+<li><tt>threatIntel</tt> : A complex JSON object representing the configuration of the threat intelligence enrichments</li>
+</ul>
+<div class="section">
+<h3><a name="The_enrichment_Configuration"></a>The <tt>enrichment</tt> Configuration</h3>
+
+<table border="0" class="table table-striped">
+  <thead>
+    
+<tr class="a">
+      
+<th>Field </th>
+      
+<th>Description </th>
+      
+<th>Example </th>
+    </tr>
+  </thead>
+  <tbody>
+    
+<tr class="b">
+      
+<td><tt>fieldToTypeMap</tt> </td>
+      
+<td>In the case of a simple HBase enrichment (i.e. a key/value lookup), the mapping between fields and the enrichment types associated with those fields must be known. This enrichment type is used as part of the HBase key. Note: applies to hbaseEnrichment only. </td>
+      
+<td><tt>&quot;fieldToTypeMap&quot; : { &quot;ip_src_addr&quot; : [ &quot;asset_enrichment&quot; ] }</tt> </td>
+    </tr>
+    
+<tr class="a">
+      
+<td><tt>fieldMap</tt> </td>
+      
+<td>The map of enrichment bolts names to configuration handlers which know how to split the message up. The simplest of which is just a list of fields. More complex examples would be the stellar enrichment which provides stellar statements. Each field listed in the array arg is sent to the enrichment referenced in the key. Cardinality of fields to enrichments is many-to-many. </td>
+      
+<td><tt>&quot;fieldMap&quot;: {&quot;hbaseEnrichment&quot;: [&quot;ip_src_addr&quot;,&quot;ip_dst_addr&quot;]}</tt> </td>
+    </tr>
+    
+<tr class="b">
+      
+<td><tt>config</tt> </td>
+      
+<td>The general configuration for the enrichment </td>
+      
+<td><tt>&quot;config&quot;: {&quot;typeToColumnFamily&quot;: { &quot;asset_enrichment&quot; : &quot;cf&quot; } }</tt> </td>
+    </tr>
+  </tbody>
+</table>
+<p>The <tt>config</tt> map is intended to house enrichment specific configuration. For instance, for the <tt>hbaseEnrichment</tt>, the mappings between the enrichment types to the column families is specified.</p>
+<p>The <tt>fieldMap</tt>contents are of interest because they contain the routing and configuration information for the enrichments.<br />When we say &#x2018;routing&#x2019;, we mean how the messages get split up and sent to the enrichment adapter bolts.<br />The simplest, by far, is just providing a simple list as in</p>
+
+<div class="source">
+<div class="source">
+<pre>    &quot;fieldMap&quot;: {
+      &quot;geo&quot;: [
+        &quot;ip_src_addr&quot;,
+        &quot;ip_dst_addr&quot;
+      ],
+      &quot;host&quot;: [
+        &quot;ip_src_addr&quot;,
+        &quot;ip_dst_addr&quot;
+      ],
+      &quot;hbaseEnrichment&quot;: [
+        &quot;ip_src_addr&quot;,
+        &quot;ip_dst_addr&quot;
+      ]
+      }
+</pre></div></div>
+<p>Based on this sample config, both <tt>ip_src_addr</tt> and <tt>ip_dst_addr</tt> will go to the <tt>geo</tt>, <tt>host</tt>, and <tt>hbaseEnrichment</tt> adapter bolts. </p>
+<div class="section">
+<h4><a name="Stellar_Enrichment_Configuration"></a>Stellar Enrichment Configuration</h4>
+<p>For the <tt>geo</tt>, <tt>host</tt> and <tt>hbaseEnrichment</tt>, this is sufficient. However, more complex enrichments may contain their own configuration. Currently, the <tt>stellar</tt> enrichment is more adaptable and thus requires a more nuanced configuration.</p>
+<p>At its most basic, we want to take a message and apply a couple of enrichments, such as converting the <tt>hostname</tt> field to lowercase. We do this by specifying the transformation inside of the <tt>config</tt> for the <tt>stellar</tt> fieldMap. There are two syntaxes that are supported, specifying the transformations as a map with the key as the field and the value the stellar expression:</p>
+
+<div class="source">
+<div class="source">
+<pre>    &quot;fieldMap&quot;: {
+       ...
+      &quot;stellar&quot; : {
+        &quot;config&quot; : {
+          &quot;hostname&quot; : &quot;TO_LOWER(hostname)&quot;
+        }
+      }
+    }
+</pre></div></div>
+<p>Another approach is to make the transformations as a list with the same <tt>var := expr</tt> syntax as is used in the Stellar REPL, such as:</p>
+
+<div class="source">
+<div class="source">
+<pre>    &quot;fieldMap&quot;: {
+       ...
+      &quot;stellar&quot; : {
+        &quot;config&quot; : [
+          &quot;hostname := TO_LOWER(hostname)&quot;
+        ]
+      }
+    }
+</pre></div></div>
+<p>Sometimes arbitrary stellar enrichments may take enough time that you would prefer to split some of them into groups and execute the groups of stellar enrichments in parallel. Take, for instance, if you wanted to do an HBase enrichment and a profiler call which were independent of one another. This usecase is supported by splitting the enrichments up as groups.</p>
+<p>Consider the following example:</p>
+
+<div class="source">
+<div class="source">
+<pre>    &quot;fieldMap&quot;: {
+       ...
+      &quot;stellar&quot; : {
+        &quot;config&quot; : {
+          &quot;malicious_domain_enrichment&quot; : {
+            &quot;is_bad_domain&quot; : &quot;ENRICHMENT_EXISTS('malicious_domains', ip_dst_addr, 'enrichments', 'cf')&quot;
+          },
+          &quot;login_profile&quot; : [
+            &quot;profile_window := PROFILE_WINDOW('from 6 months ago')&quot;, 
+            &quot;global_login_profile := PROFILE_GET('distinct_login_attempts', 'global', profile_window)&quot;,
+            &quot;stats := STATS_MERGE(global_login_profile)&quot;,
+            &quot;auth_attempts_median := STATS_PERCENTILE(stats, 0.5)&quot;, 
+            &quot;auth_attempts_sd := STATS_SD(stats)&quot;,
+            &quot;profile_window := null&quot;, 
+            &quot;global_login_profile := null&quot;, 
+            &quot;stats := null&quot;
+          ]
+        }
+      }
+    }
+</pre></div></div>
+<p>Here we want to perform two enrichments that hit HBase and we would rather not run in sequence. These enrichments are entirely independent of one another (i.e. neither relies on the output of the other). In this case, we&#x2019;ve created a group called <tt>malicious_domain_enrichment</tt> to inquire about whether the destination address exists in the HBase enrichment table in the <tt>malicious_domains</tt> enrichment type. This is a simple enrichment, so we can express the enrichment group as a map with the new field <tt>is_bad_domain</tt> being a key and the stellar expression associated with that operation being the associated value.</p>
+<p>In contrast, the stellar enrichment group <tt>login_profile</tt> is interacting with the profiler, has multiple temporary expressions (i.e. <tt>profile_window</tt>, <tt>global_login_profile</tt>, and <tt>stats</tt>) that are useful only within the context of this group of stellar expressions. In this case, we would need to ensure that we use the list construct when specifying the group and remember to set the temporary variables to <tt>null</tt> so they are not passed along.</p>
+<p>In general, things to note from this section are as follows:</p>
+
+<ul>
+  
+<li>The stellar enrichments for the <tt>stellar</tt> enrichment adapter are specified in the <tt>config</tt> for the <tt>stellar</tt> enrichment adapter in the <tt>fieldMap</tt></li>
+  
+<li>Groups of independent (i.e. no expression in any group depend on the output of an expression from an other group) may be executed in parallel</li>
+  
+<li>If you have the need to use temporary variables, you may use the list construct. Ensure that you assign the variables to <tt>null</tt> before the end of the group.</li>
+  
+<li><b>Ensure that you do not assign a field to a stellar expression which returns an object which JSON cannot represent.</b></li>
+  
+<li>Fields assigned to Maps as part of stellar enrichments have the maps unfolded, similar to the HBase Enrichment
+  
+<ul>
+    
+<li>For example the stellar enrichment for field <tt>foo</tt> which assigns a map such as <tt>foo := { 'grok' : 1, 'bar' : 'baz'}</tt>  would yield the following fields:
+    
+<ul>
+      
+<li><tt>foo.grok</tt> == <tt>1</tt></li>
+      
+<li><tt>foo.bar</tt> == <tt>'baz'</tt></li>
+    </ul></li>
+  </ul></li>
+</ul></div></div>
+<div class="section">
+<h3><a name="The_threatIntel_Configuration"></a>The <tt>threatIntel</tt> Configuration</h3>
+
+<table border="0" class="table table-striped">
+  <thead>
+    
+<tr class="a">
+      
+<th>Field </th>
+      
+<th>Description </th>
+      
+<th>Example </th>
+    </tr>
+  </thead>
+  <tbody>
+    
+<tr class="b">
+      
+<td><tt>fieldToTypeMap</tt> </td>
+      
+<td>In the case of a simple HBase threat intel enrichment (i.e. a key/value lookup), the mapping between fields and the enrichment types associated with those fields must be known. This enrichment type is used as part of the HBase key. Note: applies to hbaseThreatIntel only. </td>
+      
+<td><tt>&quot;fieldToTypeMap&quot; : { &quot;ip_src_addr&quot; : [ &quot;malicious_ips&quot; ] }</tt> </td>
+    </tr>
+    
+<tr class="a">
+      
+<td><tt>fieldMap</tt> </td>
+      
+<td>The map of threat intel enrichment bolts names to fields in the JSON messages. Each field is sent to the threat intel enrichment bolt referenced in the key. Each field listed in the array arg is sent to the enrichment referenced in the key. Cardinality of fields to enrichments is many-to-many. </td>
+      
+<td><tt>&quot;fieldMap&quot;: {&quot;hbaseThreatIntel&quot;: [&quot;ip_src_addr&quot;,&quot;ip_dst_addr&quot;]}</tt> </td>
+    </tr>
+    
+<tr class="b">
+      
+<td><tt>triageConfig</tt> </td>
+      
+<td>The configuration of the threat triage scorer. In the situation where a threat is detected, a score is assigned to the message and embedded in the indexed message. </td>
+      
+<td><tt>&quot;riskLevelRules&quot; : { &quot;IN_SUBNET(ip_dst_addr, '192.168.0.0/24')&quot; : 10 }</tt> </td>
+    </tr>
+    
+<tr class="a">
+      
+<td><tt>config</tt> </td>
+      
+<td>The general configuration for the Threat Intel </td>
+      
+<td><tt>&quot;config&quot;: {&quot;typeToColumnFamily&quot;: { &quot;malicious_ips&quot;,&quot;cf&quot; } }</tt> </td>
+    </tr>
+  </tbody>
+</table>
+<p>The <tt>config</tt> map is intended to house threat intel specific configuration. For instance, for the <tt>hbaseThreatIntel</tt> threat intel adapter, the mappings between the enrichment types to the column families is specified. The <tt>fieldMap</tt> configuration is similar to the <tt>enrichment</tt> configuration in that the adapters available are the same.</p>
+<p>The <tt>triageConfig</tt> field is also a complex field and it bears some description:</p>
+
+<table border="0" class="table table-striped">
+  <thead>
+    
+<tr class="a">
+      
+<th>Field </th>
+      
+<th>Description </th>
+      
+<th>Example </th>
+    </tr>
+  </thead>
+  <tbody>
+    
+<tr class="b">
+      
+<td><tt>riskLevelRules</tt> </td>
+      
+<td>This is a list of rules (represented as Stellar expressions) associated with scores with optional names and comments </td>
+      
+<td>see below</td>
+    </tr>
+    
+<tr class="a">
+      
+<td><tt>aggregator</tt> </td>
+      
+<td>An aggregation function that takes all non-zero scores representing the matching queries from <tt>riskLevelRules</tt> and aggregates them into a single score. </td>
+      
+<td><tt>&quot;MAX&quot;</tt> </td>
+    </tr>
+  </tbody>
+</table>
+<p>A risk level rule is of the following format:</p>
+
+<ul>
+  
+<li><tt>name</tt> : The name of the threat triage rule</li>
+  
+<li><tt>comment</tt> : A comment describing the rule</li>
+  
+<li><tt>rule</tt> : The rule, represented as a Stellar statement</li>
+  
+<li><tt>score</tt> : Associated threat triage score for the rule</li>
+  
+<li><tt>reason</tt> : Reason the rule tripped. Can be represented as a Stellar statement</li>
+</ul>
+<p>An example of a rule is as follows:</p>
+
+<div class="source">
+<div class="source">
+<pre>    &quot;riskLevelRules&quot; : [ 
+        { 
+          &quot;name&quot; : &quot;is internal&quot;
+        , &quot;comment&quot; : &quot;determines if the destination is internal.&quot;
+        , &quot;rule&quot; : &quot;IN_SUBNET(ip_dst_addr, '192.168.0.0/24')&quot;
+        , &quot;score&quot; : 10
+        , &quot;reason&quot; : &quot;FORMAT('%s is internal', ip_dst_addr)&quot;
+        }
+                       ]
+</pre></div></div>
+<p>The supported aggregation functions are:</p>
+
+<ul>
+  
+<li><tt>MAX</tt> : The max of all of the associated values for matching queries</li>
+  
+<li><tt>MIN</tt> : The min of all of the associated values for matching queries</li>
+  
+<li><tt>MEAN</tt> : The mean of all of the associated values for matching queries</li>
+  
+<li><tt>SUM</tt> : The sum of all the associated values for matching queries</li>
+  
+<li><tt>POSITIVE_MEAN</tt> : The mean of the positive associated values for the matching queries.</li>
+</ul></div>
+<div class="section">
+<h3><a name="Example_Configuration"></a>Example Configuration</h3>
+<p>An example configuration for the YAF sensor is as follows:</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+  &quot;enrichment&quot;: {
+    &quot;fieldMap&quot;: {
+      &quot;geo&quot;: [
+        &quot;ip_src_addr&quot;,
+        &quot;ip_dst_addr&quot;
+      ],
+      &quot;host&quot;: [
+        &quot;ip_src_addr&quot;,
+        &quot;ip_dst_addr&quot;
+      ],
+      &quot;hbaseEnrichment&quot;: [
+        &quot;ip_src_addr&quot;,
+        &quot;ip_dst_addr&quot;
+      ]
+    }
+  ,&quot;fieldToTypeMap&quot;: {
+      &quot;ip_src_addr&quot;: [
+        &quot;playful_classification&quot;
+      ],
+      &quot;ip_dst_addr&quot;: [
+        &quot;playful_classification&quot;
+      ]
+    }
+  },
+  &quot;threatIntel&quot;: {
+    &quot;fieldMap&quot;: {
+      &quot;hbaseThreatIntel&quot;: [
+        &quot;ip_src_addr&quot;,
+        &quot;ip_dst_addr&quot;
+      ]
+    },
+    &quot;fieldToTypeMap&quot;: {
+      &quot;ip_src_addr&quot;: [
+        &quot;malicious_ip&quot;
+      ],
+      &quot;ip_dst_addr&quot;: [
+        &quot;malicious_ip&quot;
+      ]
+    },
+    &quot;triageConfig&quot; : {
+      &quot;riskLevelRules&quot; : [ 
+        {
+          &quot;rule&quot; : &quot;ip_src_addr == '10.0.2.3' or ip_dst_addr == '10.0.2.3'&quot;,
+          &quot;score&quot; : 10
+        }
+      ],
+      &quot;aggregator&quot; : &quot;MAX&quot;
+    }
+  }
+}
+</pre></div></div>
+<p>ThreatIntel alert levels are emitted as a new field &#x201c;threat.triage.level.&#x201d; So for the example above, an incoming message that trips the <tt>ip_src_addr</tt> rule will have a new field threat.triage.level=10.</p>
+<p><a name="Example_Enrichment_via_Stellar"></a></p>
+<h1>Example Enrichment via Stellar</h1>
+<p>Let&#x2019;s walk through doing a simple enrichment using Stellar on your cluster using the Squid topology.</p></div></div>
+<div class="section">
+<h2><a name="Install_Prerequisites"></a>Install Prerequisites</h2>
+<p>Now let&#x2019;s install some prerequisites:</p>
+
+<ul>
+  
+<li>Squid client via <tt>yum install squid</tt></li>
+  
+<li>ES Head plugin via <tt>/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head</tt></li>
+</ul>
+<p>Start Squid via <tt>service squid start</tt></p></div>
+<div class="section">
+<h2><a name="Adjust_Enrichment_Configurations_for_Squid_to_Call_Stellar"></a>Adjust Enrichment Configurations for Squid to Call Stellar</h2>
+<p>Let&#x2019;s adjust the configurations for the Squid topology to annotate the messages using some Stellar functions.</p>
+
+<ul>
+  
+<li>
+<p>Edit the squid enrichment configuration at <tt>$METRON_HOME/config/zookeeper/enrichments/squid.json</tt> (this file will not exist, so create a new one) to add some new fields based on stellar queries:</p>
+  
+<div class="source">
+<div class="source">
+<pre>{
+  &quot;enrichment&quot; : {
+&quot;fieldMap&quot;: {
+  &quot;stellar&quot; : {
+    &quot;config&quot; : {
+      &quot;numeric&quot; : {
+                  &quot;foo&quot;: &quot;1 + 1&quot;
+                  }
+      ,&quot;ALL_CAPS&quot; : &quot;TO_UPPER(source.type)&quot;
+    }
+  }
+ }
+  },
+  &quot;threatIntel&quot; : {
+&quot;fieldMap&quot;:{
+ &quot;stellar&quot; : {
+    &quot;config&quot; : {
+      &quot;bar&quot; : &quot;TO_UPPER(source.type)&quot;
+    }
+  } 
+},
+&quot;triageConfig&quot; : {
+}
+  }
+}
+</pre></div></div>
+<p>We have added the following fields as part of the enrichment phase of the enrichment topology:</p></li>
+  
+<li>
+<p><tt>foo</tt> == 2</p></li>
+  
+<li><tt>ALL_CAPS</tt> == SQUID</li>
+</ul>
+<p>We have added the following as part of the threat intel:</p>
+
+<ul>
+  
+<li><tt>bar</tt> == SQUID</li>
+</ul>
+<p>Please note that foo and ALL_CAPS will be applied in separate workers due to them being in separate groups.</p>
+
+<ul>
+  
+<li>Upload new configs via <tt>$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i $METRON_HOME/config/zookeeper -z node1:2181</tt></li>
+  
+<li>Make the Squid topic in kafka via <tt>/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 --create --topic squid --partitions 1 --replication-factor 1</tt></li>
+</ul></div>
+<div class="section">
+<h2><a name="Start_Topologies_and_Send_Data"></a>Start Topologies and Send Data</h2>
+<p>Now we need to start the topologies and send some data:</p>
+
+<ul>
+  
+<li>Start the squid topology via <tt>$METRON_HOME/bin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s squid</tt></li>
+  
+<li>Generate some data via the squid client:
+  
+<ul>
+    
+<li><tt>squidclient http://yahoo.com</tt></li>
+    
+<li><tt>squidclient http://cnn.com</tt></li>
+  </ul></li>
+  
+<li>Send the data to kafka via <tt>cat /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic squid</tt></li>
+  
+<li>Browse the data in elasticsearch via the ES Head plugin @ <a class="externalLink" href="http://node1:9200/_plugin/head/">http://node1:9200/_plugin/head/</a> and verify that in the squid index you have two documents</li>
+  
+<li>Ensure that the documents have new fields <tt>foo</tt>, <tt>bar</tt> and <tt>ALL_CAPS</tt> with values as described above.</li>
+</ul>
+<p>Note that we could have used any Stellar statements here, including calling out to HBase via <tt>ENRICHMENT_GET</tt> and <tt>ENRICHMENT_EXISTS</tt> or even calling a machine learning model via <a href="../../metron-analytics/metron-maas-service/index.html">Model as a Service</a>.</p></div>
+                  </div>
+            </div>
+          </div>
+
+    <hr/>
+
+    <footer>
+            <div class="container-fluid">
+              <div class="row span12">Copyright &copy;                    2017
+                        <a href="https://www.apache.org">The Apache Software Foundation</a>.
+            All Rights Reserved.      
+                    
+      </div>
+
+                          
+        
+                </div>
+    </footer>
+  </body>
+</html>

Added: release/metron/0.4.2/site-book/metron-platform/metron-indexing/index.html
==============================================================================
--- release/metron/0.4.2/site-book/metron-platform/metron-indexing/index.html (added)
+++ release/metron/0.4.2/site-book/metron-platform/metron-indexing/index.html Wed Jan  3 18:25:57 2018
@@ -0,0 +1,555 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-12-08
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta name="Date-Revision-yyyymmdd" content="20171208" />
+    <meta http-equiv="Content-Language" content="en" />
+    <title>Metron &#x2013; Indexing</title>
+    <link rel="stylesheet" href="../../css/apache-maven-fluido-1.3.0.min.css" />
+    <link rel="stylesheet" href="../../css/site.css" />
+    <link rel="stylesheet" href="../../css/print.css" media="print" />
+
+      
+    <script type="text/javascript" src="../../js/apache-maven-fluido-1.3.0.min.js"></script>
+
+                          
+        
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel( { interval: 3500 } ) } );</script>
+          
+            </head>
+        <body class="topBarDisabled">
+          
+                
+                    
+    
+        <div class="container-fluid">
+          <div id="banner">
+        <div class="pull-left">
+                                    <a href="http://metron.apache.org/" id="bannerLeft">
+                                                                                                <img src="../../images/metron-logo.png"  alt="Apache Metron" width="148px" height="48px"/>
+                </a>
+                      </div>
+        <div class="pull-right">  </div>
+        <div class="clear"><hr/></div>
+      </div>
+
+      <div id="breadcrumbs">
+        <ul class="breadcrumb">
+                
+                    
+                              <li class="">
+                    <a href="http://www.apache.org" class="externalLink" title="Apache">
+        Apache</a>
+        </li>
+      <li class="divider ">/</li>
+            <li class="">
+                    <a href="http://metron.apache.org/" class="externalLink" title="Metron">
+        Metron</a>
+        </li>
+      <li class="divider ">/</li>
+            <li class="">
+                    <a href="../../index.html" title="Documentation">
+        Documentation</a>
+        </li>
+      <li class="divider ">/</li>
+        <li class="">Indexing</li>
+        
+                
+                    
+                  <li id="publishDate" class="pull-right">Last Published: 2017-12-08</li> <li class="divider pull-right">|</li>
+              <li id="projectVersion" class="pull-right">Version: 0.4.2</li>
+            
+                            </ul>
+      </div>
+
+            
+      <div class="row-fluid">
+        <div id="leftColumn" class="span3">
+          <div class="well sidebar-nav">
+                
+                    
+                <ul class="nav nav-list">
+                    <li class="nav-header">User Documentation</li>
+                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                                                                          
+      <li>
+    
+                          <a href="../../index.html" title="Metron">
+          <i class="icon-chevron-down"></i>
+        Metron</a>
+                    <ul class="nav nav-list">
+                      
+      <li>
+    
+                          <a href="../../Upgrading.html" title="Upgrading">
+          <i class="none"></i>
+        Upgrading</a>
+            </li>
+                                                                                                                                                      
+      <li>
+    
+                          <a href="../../metron-analytics/index.html" title="Analytics">
+          <i class="icon-chevron-right"></i>
+        Analytics</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-contrib/metron-docker/index.html" title="Docker">
+          <i class="none"></i>
+        Docker</a>
+            </li>
+                                                                                                                                                                                                                                                                                                                                                                                                            
+      <li>
+    
+                          <a href="../../metron-deployment/index.html" title="Deployment">
+          <i class="icon-chevron-right"></i>
+        Deployment</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-alerts/index.html" title="Alerts">
+          <i class="none"></i>
+        Alerts</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-config/index.html" title="Config">
+          <i class="none"></i>
+        Config</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-interface/metron-rest/index.html" title="Rest">
+          <i class="none"></i>
+        Rest</a>
+            </li>
+                                                                                                                                                                                                                                                                                              
+      <li>
+    
+                          <a href="../../metron-platform/index.html" title="Platform">
+          <i class="icon-chevron-down"></i>
+        Platform</a>
+                    <ul class="nav nav-list">
+                      
+      <li>
+    
+                          <a href="../../metron-platform/Performance-tuning-guide.html" title="Performance-tuning-guide">
+          <i class="none"></i>
+        Performance-tuning-guide</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-api/index.html" title="Api">
+          <i class="none"></i>
+        Api</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-common/index.html" title="Common">
+          <i class="none"></i>
+        Common</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-data-management/index.html" title="Data-management">
+          <i class="none"></i>
+        Data-management</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-elasticsearch/index.html" title="Elasticsearch">
+          <i class="none"></i>
+        Elasticsearch</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-enrichment/index.html" title="Enrichment">
+          <i class="none"></i>
+        Enrichment</a>
+            </li>
+                      
+      <li class="active">
+    
+            <a href="#"><i class="none"></i>Indexing</a>
+          </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-management/index.html" title="Management">
+          <i class="none"></i>
+        Management</a>
+            </li>
+                                                                        
+      <li>
+    
+                          <a href="../../metron-platform/metron-parsers/index.html" title="Parsers">
+          <i class="icon-chevron-right"></i>
+        Parsers</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-pcap-backend/index.html" title="Pcap-backend">
+          <i class="none"></i>
+        Pcap-backend</a>
+            </li>
+                      
+      <li>
+    
+                          <a href="../../metron-platform/metron-writer/index.html" title="Writer">
+          <i class="none"></i>
+        Writer</a>
+            </li>
+              </ul>
+        </li>
+                                                                                          
+      <li>
+    
+                          <a href="../../metron-sensors/index.html" title="Sensors">
+          <i class="icon-chevron-right"></i>
+        Sensors</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="../../metron-stellar/stellar-3rd-party-example/index.html" title="Stellar-3rd-party-example">
+          <i class="none"></i>
+        Stellar-3rd-party-example</a>
+            </li>
+                                                                        
+      <li>
+    
+                          <a href="../../metron-stellar/stellar-common/index.html" title="Stellar-common">
+          <i class="icon-chevron-right"></i>
+        Stellar-common</a>
+                  </li>
+                                                                                          
+      <li>
+    
+                          <a href="../../use-cases/index.html" title="Use-cases">
+          <i class="icon-chevron-right"></i>
+        Use-cases</a>
+                  </li>
+              </ul>
+        </li>
+            </ul>
+                
+                    
+                
+          <hr class="divider" />
+
+           <div id="poweredBy">
+                            <div class="clear"></div>
+                            <div class="clear"></div>
+                            <div class="clear"></div>
+                             <a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
+        <img class="builtBy" alt="Built by Maven" src="../../images/logos/maven-feather.png" />
+      </a>
+                  </div>
+          </div>
+        </div>
+        
+                
+        <div id="bodyColumn"  class="span9" >
+                                  
+            <h1>Indexing</h1>
+<p><a name="Indexing"></a></p>
+<div class="section">
+<h2><a name="Introduction"></a>Introduction</h2>
+<p>The <tt>indexing</tt> topology is a topology dedicated to taking the data from the enrichment topology that have been enriched and storing the data in one or more supported indices</p>
+
+<ul>
+  
+<li>HDFS as rolled text files, one JSON blob per line</li>
+  
+<li>Elasticsearch</li>
+  
+<li>Solr</li>
+</ul>
+<p>By default, this topology writes out to both HDFS and one of Elasticsearch and Solr.</p>
+<p>Indices are written in batch and the batch size and batch timeout are specified in the <a href="#Sensor_Indexing_Configuration">Sensor Indexing Configuration</a> via the <tt>batchSize</tt> and <tt>batchTimeout</tt> parameters. These configs are variable by sensor type.</p></div>
+<div class="section">
+<h2><a name="Indexing_Architecture"></a>Indexing Architecture</h2>
+<p><img src="../../images/indexing_arch.png" alt="Architecture" /></p>
+<p>The indexing topology is extremely simple. Data is ingested into kafka and sent to </p>
+
+<ul>
+  
+<li>An indexing bolt configured to write to either elasticsearch or Solr</li>
+  
+<li>An indexing bolt configured to write to HDFS under <tt>/apps/metron/enrichment/indexed</tt></li>
+</ul>
+<p>By default, errors during indexing are sent back into the <tt>indexing</tt> kafka queue so that they can be indexed and archived.</p></div>
+<div class="section">
+<h2><a name="Sensor_Indexing_Configuration"></a>Sensor Indexing Configuration</h2>
+<p>The sensor specific configuration is intended to configure the indexing used for a given sensor type (e.g. <tt>snort</tt>). </p>
+<p>Just like the global config, the format is a JSON stored in zookeeper and on disk at <tt>$METRON_HOME/config/zookeeper/indexing</tt>. Within the sensor-specific configuration, you can configure the individual writers. The writers currently supported are:</p>
+
+<ul>
+  
+<li><tt>elasticsearch</tt></li>
+  
+<li><tt>hdfs</tt></li>
+  
+<li><tt>solr</tt></li>
+</ul>
+<p>Depending on how you start the indexing topology, it will have either elasticsearch or solr and hdfs writers running.</p>
+<p>The configuration for an individual writer-specific configuration is a JSON map with the following fields:</p>
+
+<ul>
+  
+<li><tt>index</tt> : The name of the index to write to (defaulted to the name of the sensor).</li>
+  
+<li><tt>batchSize</tt> : The size of the batch that is written to the indices at once. Defaults to <tt>1</tt> (no batching).</li>
+  
+<li><tt>batchTimeout</tt> : The timeout after which a batch will be flushed even if batchSize has not been met. Optional. If unspecified, or set to <tt>0</tt>, it defaults to a system-determined duration which is a fraction of the Storm parameter <tt>topology.message.timeout.secs</tt>. Ignored if batchSize is <tt>1</tt>, since this disables batching.</li>
+  
+<li><tt>enabled</tt> : Whether the writer is enabled (default <tt>true</tt>).</li>
+</ul>
+<div class="section">
+<h3><a name="Meta_Alerts"></a>Meta Alerts</h3>
+<p>Alerts can be grouped, after appropriate searching, into a set of alerts called a meta alert. A meta alert is useful for maintaining the context of searching and grouping during further investigations. Standard searches can return meta alerts, but grouping and other aggregation or sorting requests will not, because there&#x2019;s not a clear way to aggregate in many cases if there are multiple alerts contained in the meta alert. All meta alerts will have the source type of metaalert, regardless of the contained alert&#x2019;s origins.</p></div>
+<div class="section">
+<h3><a name="Elasticsearch"></a>Elasticsearch</h3>
+<p>Metron comes with built-in templates for the default sensors for Elasticsearch. When adding a new sensor, it will be necessary to add a new template defining the output fields appropriately. In addition, there is a requirement for a field <tt>alert</tt> of type <tt>nested</tt> for Elasticsearch 2.x installs. This is detailed at <a href="../metron-elasticsearch/index.html#Using_Metron_with_Elasticsearch_2.x">Using Metron with Elasticsearch 2.x</a></p></div>
+<div class="section">
+<h3><a name="Indexing_Configuration_Examples"></a>Indexing Configuration Examples</h3>
+<p>For a given sensor, the following scenarios would be indicated by the following cases:</p>
+<div class="section">
+<h4><a name="Base_Case"></a>Base Case</h4>
+
+<div class="source">
+<div class="source">
+<pre>{
+}
+</pre></div></div>
+<p>or no file at all.</p>
+
+<ul>
+  
+<li>elasticsearch writer
+  
+<ul>
+    
+<li>enabled</li>
+    
+<li>batch size of 1</li>
+    
+<li>batch timeout system default</li>
+    
+<li>index name the same as the sensor</li>
+  </ul></li>
+  
+<li>hdfs writer
+  
+<ul>
+    
+<li>enabled</li>
+    
+<li>batch size of 1</li>
+    
+<li>batch timeout system default</li>
+    
+<li>index name the same as the sensor</li>
+  </ul></li>
+</ul>
+<p>If a writer config is unspecified, then a warning is indicated in the Storm console. e.g.: <tt>WARNING: Default and (likely) unoptimized writer config used for hdfs writer and sensor squid</tt></p></div>
+<div class="section">
+<h4><a name="Fully_specified"></a>Fully specified</h4>
+
+<div class="source">
+<div class="source">
+<pre>{
+   &quot;elasticsearch&quot;: {
+      &quot;index&quot;: &quot;foo&quot;,
+      &quot;batchSize&quot; : 100,
+      &quot;batchTimeout&quot; : 0,
+      &quot;enabled&quot; : true 
+    },
+   &quot;hdfs&quot;: {
+      &quot;index&quot;: &quot;foo&quot;,
+      &quot;batchSize&quot;: 1,
+      &quot;batchTimeout&quot; : 0,
+      &quot;enabled&quot; : true
+    }
+}
+</pre></div></div>
+
+<ul>
+  
+<li>elasticsearch writer
+  
+<ul>
+    
+<li>enabled</li>
+    
+<li>batch size of 100</li>
+    
+<li>batch timeout system default</li>
+    
+<li>index name of &#x201c;foo&#x201d;</li>
+  </ul></li>
+  
+<li>hdfs writer
+  
+<ul>
+    
+<li>enabled</li>
+    
+<li>batch size of 1</li>
+    
+<li>batch timeout system default</li>
+    
+<li>index name of &#x201c;foo&#x201d;</li>
+  </ul></li>
+</ul></div>
+<div class="section">
+<h4><a name="HDFS_Writer_turned_off"></a>HDFS Writer turned off</h4>
+
+<div class="source">
+<div class="source">
+<pre>{
+   &quot;elasticsearch&quot;: {
+      &quot;index&quot;: &quot;foo&quot;,
+      &quot;enabled&quot; : true 
+    },
+   &quot;hdfs&quot;: {
+      &quot;index&quot;: &quot;foo&quot;,
+      &quot;batchSize&quot;: 100,
+      &quot;batchTimeout&quot; : 0,
+      &quot;enabled&quot; : false
+    }
+}
+</pre></div></div>
+
+<ul>
+  
+<li>elasticsearch writer
+  
+<ul>
+    
+<li>enabled</li>
+    
+<li>batch size of 1</li>
+    
+<li>batch timeout system default</li>
+    
+<li>index name of &#x201c;foo&#x201d;</li>
+  </ul></li>
+  
+<li>hdfs writer
+  
+<ul>
+    
+<li>disabled</li>
+  </ul></li>
+</ul>
+<p><a name="Updates_to_Indexed_Data"></a></p>
+<h1>Updates to Indexed Data</h1>
+<p>There are clear usecases where we would want to incorporate the capability to update indexed data. Thus far, we have limited capabilities provided to support this use-case:</p>
+
+<ul>
+  
+<li>Updates to the random access index (e.g. Elasticsearch and Solr) should be supported</li>
+  
+<li>Updates to the cold storage index (e.g. HDFS) is not supported currently, however to support the batch use-case updated documents will be provided in a NoSQL write-ahead log (e.g. a HBase table) and an Java API will be provided to retrieve those updates scalably (i.e. a scan-free architecture).</li>
+</ul>
+<p>Put simply, the random access index will be always up-to-date, but the HDFS index will need to be joined to the NoSQL write-ahead log to get current updates.</p></div></div></div>
+<div class="section">
+<h2><a name="The_IndexDao_Abstraction"></a>The <tt>IndexDao</tt> Abstraction</h2>
+<p>The indices mentioned above as part of Update should be pluggable by the developer so that new write-ahead logs or real-time indices can be supported by providing an implementation supporting the data access patterns.</p>
+<p>To support a new index, one would need to implement the <tt>org.apache.metron.indexing.dao.IndexDao</tt> abstraction and provide update and search capabilities. IndexDaos may be composed and updates will be performed in parallel. This enables a flexible strategy for specifying your backing store for updates at runtime. For instance, currently the REST API supports the update functionality and may be configured with a list of IndexDao implementations to use to support the updates.</p>
+<p>Updates with the IndexDao.update method replace the current object with the new object. For partial updates, use IndexDao.patch instead.</p>
+<div class="section">
+<h3><a name="The_HBaseDao"></a>The <tt>HBaseDao</tt></h3>
+<p>Updates will be written to HBase. The key structure includes the GUID and sensor type and for each new version, a new column is created with value as the message.</p>
+<p>The HBase table and column family are configured via fields in the global configuration.</p>
+<div class="section">
+<h4><a name="update.hbase.table"></a><tt>update.hbase.table</tt></h4>
+<p>The HBase table to use for message updates.</p></div>
+<div class="section">
+<h4><a name="update.hbase.cf"></a><tt>update.hbase.cf</tt></h4>
+<p>The HBase column family to use for message updates.</p></div></div>
+<div class="section">
+<h3><a name="The_MetaAlertDao"></a>The <tt>MetaAlertDao</tt></h3>
+<p>The goal of meta alerts is to be able to group together a set of alerts while being able to transparently perform actions like searches, as if meta alerts were normal alerts. <tt>org.apache.metron.indexing.dao.MetaAlertDao</tt> extends <tt>IndexDao</tt> and enables several features: </p>
+
+<ul>
+  
+<li>the ability to get all meta alerts associated with an alert</li>
+  
+<li>creation of a meta alert</li>
+  
+<li>adding alerts to a meta alert</li>
+  
+<li>removing alerts from a meta alert</li>
+  
+<li>changing a meta alert&#x2019;s status</li>
+</ul>
+<p>The implementation of this is to denormalize the relationship between alerts and meta alerts, and store alerts as a nested field within a meta alert. The use of nested fields is to avoid the limitations of parent-child relationships (one-to-many) and merely linking by IDs (which causes issues with pagination as a result of being unable to join indices). A list of containing meta alerts is stored on an alert for the purpose of keeping source alerts and alerts contained in meta alerts in sync.</p>
+<p>The search functionality of <tt>IndexDao</tt> is wrapped by the <tt>MetaAlertDao</tt> in order to provide both regular and meta alerts side-by-side with sorting. The updating capabilities are similarly wrapped, in order to ensure updates are carried through both the alerts and associated meta alerts. Both of these functions are handled under the hood.</p>
+<p>In addition, API endpoints have been added to expose the features listed above. The denormalization handles the case of going from meta alert to alert automatically.</p>
+<p><a name="Notes_on_Performance_Tuning"></a></p>
+<h1>Notes on Performance Tuning</h1>
+<p>Default installed Metron is untuned for production deployment. By far and wide, the most likely piece to require TLC from a performance perspective is the indexing layer. An index that does not keep up will back up and you will see errors in the kafka bolt. There are a few knobs to tune to get the most out of your system.</p></div></div>
+<div class="section">
+<h2><a name="Kafka_Queue"></a>Kafka Queue</h2>
+<p>The <tt>indexing</tt> kafka queue is a collection point from the enrichment topology. As such, make sure that the number of partitions in the kafka topic is sufficient to handle the throughput that you expect.</p></div>
+<div class="section">
+<h2><a name="Indexing_Topology"></a>Indexing Topology</h2>
+<p>The <tt>indexing</tt> topology as started by the <tt>$METRON_HOME/bin/start_elasticsearch_topology.sh</tt> or <tt>$METRON_HOME/bin/start_solr_topology.sh</tt> script uses a default of one executor per bolt. In a real production system, this should be customized by modifying the flux file in <tt>$METRON_HOME/flux/indexing/remote.yaml</tt>. </p>
+
+<ul>
+  
+<li>Add a <tt>parallelism</tt> field to the bolts to give Storm a parallelism  hint for the various components. Give bolts which appear to be bottlenecks (e.g. the indexing bolt) a larger hint.</li>
+  
+<li>Add a <tt>parallelism</tt> field to the kafka spout which matches the number of partitions for the enrichment kafka queue.</li>
+  
+<li>Adjust the number of workers for the topology by adjusting the  <tt>topology.workers</tt> field for the topology.</li>
+</ul>
+<p>Finally, if workers and executors are new to you or you don&#x2019;t know where to modify the flux file, the following might be of use to you:</p>
+
+<ul>
+  
+<li><a class="externalLink" href="http://www.michael-noll.com/blog/2012/10/16/understanding-the-parallelism-of-a-storm-topology/">Understanding the Parallelism of a Storm Topology</a></li>
+  
+<li><a class="externalLink" href="http://storm.apache.org/releases/current/flux.html">Flux Docs</a></li>
+</ul></div>
+<div class="section">
+<h2><a name="Zeppelin_Notebooks"></a>Zeppelin Notebooks</h2>
+<p>Zeppelin notebooks can be added to <tt>/src/main/config/zeppelin/</tt> (and subdirectories can be created for organization). The placed files must be .json files and be named appropriately. These files must be added to the metron.spec file and the RPMs rebuilt to be available to be loaded into Ambari.</p>
+<p>The notebook files will be found on the server in <tt>$METRON_HOME/config/zeppelin</tt></p>
+<p>The Ambari Management Pack has a custom action to load these templates, ZEPPELIN_DASHBOARD_INSTALL, that will import them into Zeppelin.</p></div>
+                  </div>
+            </div>
+          </div>
+
+    <hr/>
+
+    <footer>
+            <div class="container-fluid">
+              <div class="row span12">Copyright &copy;                    2017
+                        <a href="https://www.apache.org">The Apache Software Foundation</a>.
+            All Rights Reserved.      
+                    
+      </div>
+
+                          
+        
+                </div>
+    </footer>
+  </body>
+</html>



Mime
View raw message