metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ceste...@apache.org
Subject [15/15] incubator-metron git commit: METRON-766: Release 0.3.1 closes apache/incubator-metron#477
Date Fri, 17 Mar 2017 14:51:36 GMT
METRON-766: Release 0.3.1 closes apache/incubator-metron#477


Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/a055de44
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/a055de44
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/a055de44

Branch: refs/heads/asf-site
Commit: a055de442b3862f8328ccea966ba7996995318f2
Parents: db6c9f0
Author: cstella <cestella@gmail.com>
Authored: Fri Mar 17 10:51:09 2017 -0400
Committer: cstella <cestella@gmail.com>
Committed: Fri Mar 17 10:51:09 2017 -0400

----------------------------------------------------------------------
 current-book/Upgrading.html                     |  293 +
 .../css/apache-maven-fluido-1.3.0.min.css       |    9 +
 current-book/css/print.css                      |   23 +
 current-book/css/site.css                       |    1 +
 current-book/images/ApacheIncubating_Logo.png   |  Bin 0 -> 11294 bytes
 current-book/images/accessories-text-editor.png |  Bin 0 -> 746 bytes
 current-book/images/add.gif                     |  Bin 0 -> 397 bytes
 current-book/images/apache-maven-project-2.png  |  Bin 0 -> 33442 bytes
 current-book/images/application-certificate.png |  Bin 0 -> 923 bytes
 current-book/images/contact-new.png             |  Bin 0 -> 736 bytes
 current-book/images/document-properties.png     |  Bin 0 -> 577 bytes
 current-book/images/drive-harddisk.png          |  Bin 0 -> 700 bytes
 current-book/images/enrichment_arch.png         |  Bin 0 -> 113606 bytes
 current-book/images/fix.gif                     |  Bin 0 -> 366 bytes
 current-book/images/icon_error_sml.gif          |  Bin 0 -> 633 bytes
 current-book/images/icon_help_sml.gif           |  Bin 0 -> 1072 bytes
 current-book/images/icon_info_sml.gif           |  Bin 0 -> 638 bytes
 current-book/images/icon_success_sml.gif        |  Bin 0 -> 604 bytes
 current-book/images/icon_warning_sml.gif        |  Bin 0 -> 625 bytes
 current-book/images/image-x-generic.png         |  Bin 0 -> 662 bytes
 current-book/images/indexing_arch.png           |  Bin 0 -> 86115 bytes
 current-book/images/internet-web-browser.png    |  Bin 0 -> 1017 bytes
 .../images/logos/build-by-maven-black.png       |  Bin 0 -> 2294 bytes
 .../images/logos/build-by-maven-white.png       |  Bin 0 -> 2260 bytes
 current-book/images/logos/maven-feather.png     |  Bin 0 -> 3330 bytes
 current-book/images/maas_arch.png               |  Bin 0 -> 106286 bytes
 current-book/images/metron-logo.png             |  Bin 0 -> 21186 bytes
 current-book/images/network-server.png          |  Bin 0 -> 536 bytes
 current-book/images/package-x-generic.png       |  Bin 0 -> 717 bytes
 current-book/images/parser_arch.png             |  Bin 0 -> 81891 bytes
 current-book/images/profiles/pre-release.png    |  Bin 0 -> 32607 bytes
 current-book/images/profiles/retired.png        |  Bin 0 -> 22003 bytes
 current-book/images/profiles/sandbox.png        |  Bin 0 -> 33010 bytes
 current-book/images/remove.gif                  |  Bin 0 -> 607 bytes
 current-book/images/rss.png                     |  Bin 0 -> 474 bytes
 current-book/images/update.gif                  |  Bin 0 -> 1090 bytes
 current-book/images/window-new.png              |  Bin 0 -> 583 bytes
 current-book/img/glyphicons-halflings-white.png |  Bin 0 -> 8777 bytes
 current-book/img/glyphicons-halflings.png       |  Bin 0 -> 12799 bytes
 current-book/index.html                         |  257 +
 .../js/apache-maven-fluido-1.3.0.min.js         |   21 +
 current-book/metron-analytics/index.html        |  210 +
 .../metron-maas-service/index.html              |  474 ++
 .../metron-profiler-client/index.html           |  427 +
 .../metron-analytics/metron-profiler/index.html |  779 ++
 .../metron-statistics/HLLP.html                 | 7587 ++++++++++++++++++
 .../metron-statistics/index.html                |  916 +++
 .../metron-deployment/amazon-ec2/index.html     |  521 ++
 current-book/metron-deployment/index.html       |  434 +
 .../packaging/docker/ansible-docker/index.html  |  239 +
 .../packaging/docker/rpm-docker/index.html      |  226 +
 .../metron-deployment/packer-build/index.html   |  274 +
 current-book/metron-deployment/roles/index.html |  271 +
 .../metron-deployment/roles/kibana/index.html   |  285 +
 .../metron-deployment/roles/monit/index.html    |  327 +
 .../roles/opentaxii/index.html                  |  419 +
 .../roles/pcap_replay/index.html                |  294 +
 .../roles/sensor-stubs/index.html               |  340 +
 .../roles/sensor-test-mode/index.html           |  284 +
 .../vagrant/codelab-platform/index.html         |  315 +
 .../vagrant/fastcapa-test-platform/index.html   |  313 +
 .../vagrant/full-dev-platform/index.html        |  374 +
 .../metron-deployment/vagrant/index.html        |  255 +
 .../vagrant/quick-dev-platform/index.html       |  327 +
 current-book/metron-docker/index.html           |  470 ++
 current-book/metron-platform/index.html         |  249 +
 .../metron-platform/metron-api/index.html       |  285 +
 .../metron-platform/metron-common/index.html    | 2233 ++++++
 .../metron-data-management/index.html           | 1014 +++
 .../metron-enrichment/index.html                |  670 ++
 .../metron-platform/metron-indexing/index.html  |  428 +
 .../metron-management/index.html                | 1602 ++++
 .../metron-platform/metron-parsers/index.html   |  691 ++
 .../metron-pcap-backend/index.html              |  376 +
 current-book/metron-sensors/fastcapa/index.html |  195 +
 current-book/metron-sensors/index.html          |  193 +
 current-book/metron-sensors/pycapa/index.html   |  262 +
 develop/bylaws/index.html                       |  514 --
 develop/coding/index.html                       |  249 -
 documentation/index.html                        |   22 +-
 feed.xml                                        |    4 +-
 81 files changed, 25181 insertions(+), 771 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a055de44/current-book/Upgrading.html
----------------------------------------------------------------------
diff --git a/current-book/Upgrading.html b/current-book/Upgrading.html
new file mode 100644
index 0000000..0e083cf
--- /dev/null
+++ b/current-book/Upgrading.html
@@ -0,0 +1,293 @@
+<!DOCTYPE html>
+<!--
+ | Generated by Apache Maven Doxia at 2017-02-23
+ | Rendered using Apache Maven Fluido Skin 1.3.0
+-->
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta name="Date-Revision-yyyymmdd" content="20170223" />
+    <meta http-equiv="Content-Language" content="en" />
+    <title>Metron &#x2013; Upgrading</title>
+    <link rel="stylesheet" href="./css/apache-maven-fluido-1.3.0.min.css" />
+    <link rel="stylesheet" href="./css/site.css" />
+    <link rel="stylesheet" href="./css/print.css" media="print" />
+
+      
+    <script type="text/javascript" src="./js/apache-maven-fluido-1.3.0.min.js"></script>
+
+                          
+        
+<script type="text/javascript">$( document ).ready( function() { $( '.carousel' ).carousel(
{ interval: 3500 } ) } );</script>
+          
+            </head>
+        <body class="topBarDisabled">
+          
+                
+                    
+    
+        <div class="container-fluid">
+          <div id="banner">
+        <div class="pull-left">
+                                    <a href="http://metron.incubator.apache.org/" id="bannerLeft">
+                                                                                        
       <img src="images/metron-logo.png"  alt="Apache Metron - Incubating" width="148px"
height="48px"/>
+                </a>
+                      </div>
+        <div class="pull-right">                  <a href="http://incubator.apache.org/"
id="bannerRight">
+                                                                                        
       <img src="images/ApacheIncubating_Logo.png"  alt="Apache Incubating" width="192px"
height="48px"/>
+                </a>
+      </div>
+        <div class="clear"><hr/></div>
+      </div>
+
+      <div id="breadcrumbs">
+        <ul class="breadcrumb">
+                
+                    
+                              <li class="">
+                    <a href="http://www.apache.org" class="externalLink" title="Apache">
+        Apache</a>
+        </li>
+      <li class="divider ">/</li>
+            <li class="">
+                    <a href="http://metron.incubator.apache.org/" class="externalLink"
title="Metron-Incubating">
+        Metron-Incubating</a>
+        </li>
+      <li class="divider ">/</li>
+            <li class="">
+                    <a href="index.html" title="Documentation">
+        Documentation</a>
+        </li>
+      <li class="divider ">/</li>
+        <li class="">Upgrading</li>
+        
+                
+                    
+                  <li id="publishDate" class="pull-right">Last Published: 2017-02-23</li>
<li class="divider pull-right">|</li>
+              <li id="projectVersion" class="pull-right">Version: 0.3.1</li>
+            
+                            </ul>
+      </div>
+
+            
+      <div class="row-fluid">
+        <div id="leftColumn" class="span3">
+          <div class="well sidebar-nav">
+                
+                    
+                <ul class="nav nav-list">
+                    <li class="nav-header">User Documentation</li>
+                                                                                        
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                                                         
                                                         
+      <li>
+    
+                          <a href="index.html" title="Metron">
+          <i class="icon-chevron-down"></i>
+        Metron</a>
+                    <ul class="nav nav-list">
+                      
+      <li class="active">
+    
+            <a href="#"><i class="none"></i>Upgrading</a>
+          </li>
+                                                                                        
                                                             
+      <li>
+    
+                          <a href="metron-analytics/index.html" title="Analytics">
+          <i class="icon-chevron-right"></i>
+        Analytics</a>
+                  </li>
+                                                                                        
                                                                                         
                                                                                         
                                                                                     
+      <li>
+    
+                          <a href="metron-deployment/index.html" title="Deployment">
+          <i class="icon-chevron-right"></i>
+        Deployment</a>
+                  </li>
+                      
+      <li>
+    
+                          <a href="metron-docker/index.html" title="Docker">
+          <i class="none"></i>
+        Docker</a>
+            </li>
+                                                                                        
                                                                                         
                   
+      <li>
+    
+                          <a href="metron-platform/index.html" title="Platform">
+          <i class="icon-chevron-right"></i>
+        Platform</a>
+                  </li>
+                                                                                        
 
+      <li>
+    
+                          <a href="metron-sensors/index.html" title="Sensors">
+          <i class="icon-chevron-right"></i>
+        Sensors</a>
+                  </li>
+              </ul>
+        </li>
+            </ul>
+                
+                    
+                
+          <hr class="divider" />
+
+           <div id="poweredBy">
+                            <div class="clear"></div>
+                            <div class="clear"></div>
+                            <div class="clear"></div>
+                             <a href="http://maven.apache.org/" title="Built by Maven"
class="poweredBy">
+        <img class="builtBy" alt="Built by Maven" src="./images/logos/maven-feather.png"
/>
+      </a>
+                  </div>
+          </div>
+        </div>
+        
+                
+        <div id="bodyColumn"  class="span9" >
+                                  
+            <h1>Upgrading</h1>
+<p><a name="Upgrading"></a></p>
+<p>This document constitutes a per-version listing of changes of configuration which
are non-backwards compatible.</p>
+<div class="section">
+<h2><a name="a0.2.0BETA_to_0.3.0"></a>0.2.0BETA to 0.3.0</h2>
+<div class="section">
+<h3><a name="METRON-447:_Monit_fails_to_reload_when_upgrading_from_0.2.0BETA_to_master"></a><a
class="externalLink" href="https://issues.apache.org/jira/browse/METRON-447">METRON-447:
Monit fails to reload when upgrading from 0.2.0BETA to master</a></h3>
+<div class="section">
+<h4><a name="Description"></a>Description</h4>
+<p><tt>/etc/monit.d/enrichment-elasticsearch.monit</tt> was renamed to
<tt>/etc/monit.d/indexing-elasticsearch.monit</tt>, however the old file isn&#x2019;t
removed via ansible, which causes the below error during an upgrade: <tt>Starting monit:
/etc/monit.d/enrichment-elasticsearch.monit:18: Service
+name conflict, enrichment already defined
+'/usr/local/monit/status_enrichment_topology.sh'</tt></p></div></div>
+<div class="section">
+<h3><a name="METRON-448:Upgrading_via_Ansible_deployment_does_not_add_topology.classpath"></a><a
class="externalLink" href="https://issues.apache.org/jira/browse/METRON-448">METRON-448:Upgrading
via Ansible deployment does not add topology.classpath </a></h3>
+<div class="section">
+<h4><a name="Description"></a>Description</h4>
+<p>When using Ansible to deploy the latest Metron bits to an existing installation,
storm-site is not being updated with the new 0.2.1BETA parameter <tt>topology.classpath</tt>.
Topologies are unable to find the client configs as a result.</p></div>
+<div class="section">
+<h4><a name="Workaround"></a>Workaround</h4>
+<p>Set the <tt>topology.classpath</tt> property for storm in Ambari to
<tt>/etc/hbase/conf:/etc/hadoop/conf</tt></p></div></div></div>
+<div class="section">
+<h2><a name="a0.3.0_to_0.3.1"></a>0.3.0 to 0.3.1</h2>
+<div class="section">
+<h3><a name="METRON-664:_Make_the_index_configuration_per-writer_with_enableddisabled"></a><a
class="externalLink" href="https://issues.apache.org/jira/browse/METRON-664">METRON-664:
Make the index configuration per-writer with enabled/disabled</a></h3>
+<div class="section">
+<h4><a name="Description"></a>Description</h4>
+<p>As of 0.3.0 the indexing configuration</p>
+
+<ul>
+  
+<li>Is held in the enrichment configuration for a sensor</li>
+  
+<li>Has properties which control every writers (i.e. HDFS, solr or elasticsearch).</li>
+</ul>
+<p>In the 0.3.1 release, this configuration has been broken out and control for individual
writers are separated.</p>
+<p>Please see the description of the configurations in the indexing <a class="externalLink"
href="https://github.com/apache/incubator-metron/tree/Metron_0.3.1/metron-platform/metron-indexing#sensor-indexing-configuration">README</a></p></div>
+<div class="section">
+<h4><a name="Migration"></a>Migration</h4>
+<p>Migrate the configurations from each sensor enrichment configuration and create
appropriate configurations for indexing.</p>
+<p>For instance, if a sensor enrichment config for sensor <tt>foo</tt>
is in <tt>$METRON_HOME/config/zookeeper/enrichments/foo.json</tt> and looks like</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+  &quot;index&quot; : &quot;foo&quot;,
+  &quot;batchSize&quot; : 100
+}
+</pre></div></div>
+<p>You would create a file to configure each writer for sensor <tt>foo</tt>
called <tt>$METRON_HOME/config/zookeeper/indexing/foo.json</tt> with the contents</p>
+
+<div class="source">
+<div class="source">
+<pre>{
+  &quot;elasticsearch&quot; : {
+    &quot;index&quot; : &quot;foo&quot;,
+    &quot;batchSize&quot; : 100,
+    &quot;enabled&quot; : true
+  },
+  &quot;hdfs&quot; : { 
+    &quot;index&quot; : &quot;foo&quot;,
+    &quot;batchSize&quot; : 100,
+    &quot;enabled&quot; : true
+  }
+}
+</pre></div></div></div></div>
+<div class="section">
+<h3><a name="METRON-675:_Make_Threat_Triage_rules_able_to_be_assigned_names_and_comments"></a><a
class="externalLink" href="https://issues.apache.org/jira/browse/METRON-675">METRON-675:
Make Threat Triage rules able to be assigned names and comments</a></h3>
+<div class="section">
+<h4><a name="Description"></a>Description</h4>
+<p>As of 0.3.0, threat triage rules were defined as a simple Map associating a Stellar
expression with a score. As of 0.3.1, due to the fact that there may be many threat triage
rules, we have made the rules more complex. To help organize these, we have made the threat
triage objects in their own right that contain optional name and optional comment fields.</p>
+<p>This essentially makes the risk level rules slightly more complex. The format goes
from:</p>
+
+<div class="source">
+<div class="source">
+<pre>&quot;riskLevelRules&quot; : {
+    &quot;stellar expression&quot; : numeric score
+}
+</pre></div></div>
+<p>to:</p>
+
+<div class="source">
+<div class="source">
+<pre>&quot;riskLevelRules&quot; : [
+     {
+        &quot;name&quot; : &quot;optional name&quot;,
+        &quot;comment&quot; : &quot;optional comment&quot;,
+        &quot;rule&quot; : &quot;stellar expression&quot;,
+        &quot;score&quot; : numeric score
+     }
+]
+</pre></div></div></div>
+<div class="section">
+<h4><a name="Migration"></a>Migration</h4>
+<p>For every sensor enrichment configuration, you will need to migrate the <tt>riskLevelRules</tt>
section to move from a map to a list of risk level rule objects.</p></div></div>
+<div class="section">
+<h3><a name="METRON-283:_Migrate_Geo_Enrichment_outside_of_MySQL"></a><a
class="externalLink" href="https://issues.apache.org/jira/browse/METRON-283">METRON-283:
Migrate Geo Enrichment outside of MySQL</a></h3>
+<div class="section">
+<h4><a name="Description"></a>Description</h4>
+<p>As of 0.3.0, a MySQL database was used for storage and retrieval of GeoIP information
during enrichment. As of 0.3.1, the MySQL database is removed in favor of using MaxMind&#x2019;s
binary GeoIP files and stored on HDFS</p>
+<p>After initial setup, this change is transparent and existing enrichment definitions
will run as-is.</p></div>
+<div class="section">
+<h4><a name="Migration"></a>Migration</h4>
+<p>While new installs will not require any additional steps, in an existing install
a script must be run to retrieve and load the initial data.</p>
+<p>The shell script <tt>geo_enrichment_load.sh</tt> will retrieve MaxMind
GeoLite2 data and load data into HDFS, and update the configuration to point to this data.
In most cases the following usage will grab the data appropriately:</p>
+
+<div class="source">
+<div class="source">
+<pre>$METRON_HOME/bin/geo_enrichment_load.sh -z &lt;zk_server&gt;:&lt;zk_port&gt;
+</pre></div></div>
+<p>Additional options, including changing the source file location (which can be a
file:// location if the GeoIP data is already downloaded), are available with the -h flag
and are also detailed in the metron-data-management README.me  file.</p>
+<p>One caveat is that this script will NOT update on disk config files. It is recommended
to retrieve the configuration using</p>
+
+<div class="source">
+<div class="source">
+<pre>$METRON_HOME/bin/zk_load_configs.sh -z &lt;zk_server&gt;:&lt;zk_port&gt;
-m DUMP
+</pre></div></div>
+<p>The new config will be <tt>geo.hdfs.file</tt> in the global section
of the configuration. Append this key-value into the global.json in the config directory.
A PUSH is unnecessary</p></div></div>
+<div class="section">
+<h3><a name="METRON-684:_Decouple_Timestamp_calculation_from_PROFILE_GET"></a><a
class="externalLink" href="https://issues.apache.org/jira/browse/METRON-684">METRON-684:
Decouple Timestamp calculation from PROFILE_GET</a></h3>
+<div class="section">
+<h4><a name="Description"></a>Description</h4>
+<p>During 0.3.1 we decoupled specifying durations for calls to the profiler into a
separate function. The consequence is that existing calls to <tt>PROFILE_GET</tt>
will need to migrate.</p></div>
+<div class="section">
+<h4><a name="Migration"></a>Migration</h4>
+<p>Existing calls to <tt>PROFILE_GET</tt> will need to change from <tt>PROFILE_GET('profile',
'entity', duration, 'durationUnits')</tt> to <tt>PROFILE_GET('profile', 'entity',
PROFILE_FIXED(duration, 'durationUnits'))</tt></p></div></div></div>
+                  </div>
+            </div>
+          </div>
+
+    <hr/>
+
+    <footer>
+            <div class="container-fluid">
+              <div class="row span12">Copyright &copy;                   2017.
+          All Rights Reserved.      
+                    
+      </div>
+
+                          
+        
+                </div>
+    </footer>
+  </body>
+</html>


Mime
View raw message