metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ceste...@apache.org
Subject incubator-metron git commit: METRON-752 Fix documentation typos and MD issues closes apache/incubator-metron#470
Date Thu, 09 Mar 2017 18:50:51 GMT
Repository: incubator-metron
Updated Branches:
  refs/heads/master b2e46f574 -> c793308cb


METRON-752 Fix documentation typos and MD issues closes apache/incubator-metron#470


Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/c793308c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/c793308c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/c793308c

Branch: refs/heads/master
Commit: c793308cbd44e4cdbc223b141ed89978d8a52acb
Parents: b2e46f5
Author: JonZeolla <zeolla@gmail.com>
Authored: Thu Mar 9 13:50:41 2017 -0500
Committer: cstella <cestella@gmail.com>
Committed: Thu Mar 9 13:50:41 2017 -0500

----------------------------------------------------------------------
 Upgrading.md                                    |  40 +++----
 metron-analytics/metron-profiler/README.md      |  48 ++++----
 metron-interface/metron-rest/README.md          | 110 +++++++++----------
 .../metron-data-management/README.md            |  66 +++++------
 site-book/bin/fix-md-dialect.py                 |  16 +--
 5 files changed, 140 insertions(+), 140 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/Upgrading.md
----------------------------------------------------------------------
diff --git a/Upgrading.md b/Upgrading.md
index 2a2ea50..92e958b 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -2,26 +2,6 @@
 This document constitutes a per-version listing of changes of
 configuration which are non-backwards compatible.
 
-## 0.2.0BETA to 0.3.0
-### [METRON-447: Monit fails to reload when upgrading from 0.2.0BETA to master](https://issues.apache.org/jira/browse/METRON-447)
-
-#### Description
-
-`/etc/monit.d/enrichment-elasticsearch.monit` was renamed to
-`/etc/monit.d/indexing-elasticsearch.monit`, however the old file isn't
-removed via ansible, which causes the below error during an upgrade:
-`Starting monit: /etc/monit.d/enrichment-elasticsearch.monit:18: Service
-name conflict, enrichment already defined
-'/usr/local/monit/status_enrichment_topology.sh'`
-
-### [METRON-448:Upgrading via Ansible deployment does not add topology.classpath ](https://issues.apache.org/jira/browse/METRON-448)
-
-#### Description
-When using Ansible to deploy the latest Metron bits to an existing installation, storm-site
is not being updated with the new 0.2.1BETA parameter `topology.classpath`. Topologies are
unable to find the client configs as a result.
-
-#### Workaround
-Set the `topology.classpath` property for storm in Ambari to `/etc/hbase/conf:/etc/hadoop/conf`
-
 ## 0.3.0 to 0.3.1
 
 ### [METRON-664: Make the index configuration per-writer with enabled/disabled](https://issues.apache.org/jira/browse/METRON-664)
@@ -152,3 +132,23 @@ into a separate function.  The consequence is that existing calls to
 
 Existing calls to `PROFILE_GET` will need to change from `PROFILE_GET('profile', 'entity',
duration, 'durationUnits')` to `PROFILE_GET('profile', 'entity', PROFILE_FIXED(duration, 'durationUnits'))`
 
+## 0.2.0BETA to 0.3.0
+### [METRON-447: Monit fails to reload when upgrading from 0.2.0BETA to master](https://issues.apache.org/jira/browse/METRON-447)
+
+#### Description
+
+`/etc/monit.d/enrichment-elasticsearch.monit` was renamed to
+`/etc/monit.d/indexing-elasticsearch.monit`, however the old file isn't
+removed via ansible, which causes the below error during an upgrade:
+`Starting monit: /etc/monit.d/enrichment-elasticsearch.monit:18: Service
+name conflict, enrichment already defined
+'/usr/local/monit/status_enrichment_topology.sh'`
+
+### [METRON-448:Upgrading via Ansible deployment does not add topology.classpath ](https://issues.apache.org/jira/browse/METRON-448)
+
+#### Description
+When using Ansible to deploy the latest Metron bits to an existing installation, storm-site
is not being updated with the new 0.2.1BETA parameter `topology.classpath`. Topologies are
unable to find the client configs as a result.
+
+#### Workaround
+Set the `topology.classpath` property for storm in Ambari to `/etc/hbase/conf:/etc/hadoop/conf`
+

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/metron-analytics/metron-profiler/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler/README.md b/metron-analytics/metron-profiler/README.md
index b4fc104..bb9f530 100644
--- a/metron-analytics/metron-profiler/README.md
+++ b/metron-analytics/metron-profiler/README.md
@@ -69,7 +69,6 @@ More information on configuring and using the client can be found [here](../metr
 It is assumed that the `PROFILE_GET` client is correctly configured before using it.
     ```
     $ bin/stellar -z node1:2181
-    
     [Stellar]>>> PROFILE_GET( "test", "10.0.0.1", PROFILE_FIXED(30, "MINUTES"))
     [451, 448]
     ```
@@ -80,15 +79,15 @@ The Profiler specification requires a JSON-formatted set of elements,
many of wh
 The specification for the Profiler topology is stored in Zookeeper at  `/metron/topology/profiler`.
 These properties also exist in the local filesystem at `$METRON_HOME/config/zookeeper/profiler.json`.

 The values can be changed on disk and then uploaded to Zookeeper using `$METRON_HOME/bin/zk_load_configs.sh`.
 
-| Name 	                        |               | Description 	
-|---	                        |---	        |---
-| [profile](#profile)           | Required   	| Unique name identifying the profile. 
-| [foreach](#foreach)           | Required  	| A separate profile is maintained "for each"
of these. 
-| [onlyif](#onlyif)  	        | Optional  	| Boolean expression that determines if a message
should be applied to the profile.
+| Name                          |               | Description
+|---                            |---            |---
+| [profile](#profile)           | Required      | Unique name identifying the profile. 
+| [foreach](#foreach)           | Required      | A separate profile is maintained "for each"
of these. 
+| [onlyif](#onlyif)             | Optional      | Boolean expression that determines if a
message should be applied to the profile.
 | [groupBy](#groupby)           | Optional      | One or more Stellar expressions used to
group the profile measurements when persisted.
-| [init](#init)  	            | Optional  	| One or more expressions executed at the start
of a window period.
-| [update](#update)  	        | Required  	| One or more expressions executed when a message
is applied to the profile.
-| [result](#result)   	        | Required  	| Stellar expressions that are executed when
the window period expires.
+| [init](#init)                 | Optional      | One or more expressions executed at the
start of a window period.
+| [update](#update)             | Required      | One or more expressions executed when a
message is applied to the profile.
+| [result](#result)             | Required      | Stellar expressions that are executed when
the window period expires.
 | [expires](#expires)           | Optional      | Profile data is purged after this period
of time, specified in milliseconds.
 
 ### `profile` 
@@ -174,7 +173,6 @@ A required Stellar expression that results in a value that is persisted
in the p
 An alternative, simplified form is also acceptable.
 ```
 "result": "2 + 2"
-
 ```
 
 **triage**
@@ -204,21 +202,21 @@ The Profiler runs as an independent Storm topology.  The configuration
for the P
 The values can be changed on disk and then the Profiler topology must be restarted.
 
 
-| Setting   | Description   |
-|---        |---            |
-| profiler.workers | The number of worker processes to create for the topology.   |
-| profiler.executors | The number of executors to spawn per component.  |
-| profiler.input.topic | The name of the Kafka topic from which to consume data.  |
-| profiler.output.topic | The name of the Kafka topic to which profile data is written. 
Only used with profiles that use the [`triage` result field](#result).  |
-| profiler.period.duration | The duration of each profile period.  This value should be defined
along with `profiler.period.duration.units`.  |
-| profiler.period.duration.units | The units used to specify the `profiler.period.duration`.
|
-| profiler.ttl | If a message has not been applied to a Profile in this period of time, the
Profile will be forgotten and its resources will be cleaned up. This value should be defined
along with `profiler.ttl.units`. |
-| profiler.ttl.units | The units used to specify the `profiler.ttl`. |
-| profiler.hbase.salt.divisor  |  A salt is prepended to the row key to help prevent hotspotting.
 This constant is used to generate the salt.  Ideally, this constant should be roughly equal
to the number of nodes in the Hbase cluster.  |
-| profiler.hbase.table | The name of the HBase table that profiles are written to.  |
-| profiler.hbase.column.family | The column family used to store profiles. |
-| profiler.hbase.batch | The number of puts that are written in a single batch.  |
-| profiler.hbase.flush.interval.seconds | The maximum number of seconds between batch writes
to HBase. |
+| Setting                               | Description
+|---                                    |---
+| profiler.workers                      | The number of worker processes to create for the
topology.
+| profiler.executors                    | The number of executors to spawn per component.
+| profiler.input.topic                  | The name of the Kafka topic from which to consume
data.
+| profiler.output.topic                 | The name of the Kafka topic to which profile data
is written.  Only used with profiles that use the [`triage` result field](#result).
+| profiler.period.duration              | The duration of each profile period.  This value
should be defined along with `profiler.period.duration.units`.
+| profiler.period.duration.units        | The units used to specify the `profiler.period.duration`.
+| profiler.ttl                          | If a message has not been applied to a Profile
in this period of time, the Profile will be forgotten and its resources will be cleaned up.
This value should be defined along with `profiler.ttl.units`.
+| profiler.ttl.units                    | The units used to specify the `profiler.ttl`.
+| profiler.hbase.salt.divisor           | A salt is prepended to the row key to help prevent
hotspotting.  This constant is used to generate the salt.  Ideally, this constant should be
roughly equal to the number of nodes in the Hbase cluster.
+| profiler.hbase.table                  | The name of the HBase table that profiles are written
to.
+| profiler.hbase.column.family          | The column family used to store profiles.
+| profiler.hbase.batch                  | The number of puts that are written in a single
batch.
+| profiler.hbase.flush.interval.seconds | The maximum number of seconds between batch writes
to HBase.
 
 After altering the configuration, start the Profiler.
 

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/metron-interface/metron-rest/README.md
----------------------------------------------------------------------
diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md
index cf41bf3..8de1982 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -10,24 +10,24 @@ This module provides a RESTful API for interacting with Metron.
 
 ## Installation
 1. Package the application with Maven:
-```
-mvn clean package
-```
+  ```
+  mvn clean package
+  ```
 
 1. Untar the archive in the target directory.  The directory structure will look like:
-```
-bin
-  start_metron_rest.sh
-lib
-  metron-rest-$METRON_VERSION.jar
-```
+  ```
+  bin
+    start_metron_rest.sh
+  lib
+    metron-rest-$METRON_VERSION.jar
+  ```
 
-1. Create an `application.yml` file with the contents of [application-docker.yml](src/main/resources/application-docker.yml).
 Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc) in properties
containing `${docker.host.address}` and update the `spring.datasource.*` properties as needed
(see the [Security](#security) section for more details).
+1. Create an `application.yml` file with the contents of [application-docker.yml](src/main/resources/application-docker.yml).
 Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc.) in properties
containing `${docker.host.address}` and update the `spring.datasource.*` properties as needed
(see the [Security](#security) section for more details).
 
 1. Start the application with this command:
-```
-./bin/start_metron_rest.sh /path/to/application.yml
-```
+  ```
+  ./bin/start_metron_rest.sh /path/to/application.yml
+  ```
 
 ## Usage
 
@@ -45,36 +45,36 @@ For [production use](http://docs.spring.io/spring-boot/docs/1.4.1.RELEASE/refere
 1. Create a MySQL user for the Metron REST application (http://dev.mysql.com/doc/refman/5.7/en/adding-users.html).
 
 1. Connect to MySQL and create a Metron REST database:
-```
-CREATE DATABASE IF NOT EXISTS metronrest
-```
+  ```
+  CREATE DATABASE IF NOT EXISTS metronrest
+  ```
 
 1. Add users:
-```
-use metronrest;
-insert into users (username, password, enabled) values ('your_username','your_password',1);
-insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
-```
+  ```
+  use metronrest;
+  insert into users (username, password, enabled) values ('your_username','your_password',1);
+  insert into authorities (username, authority) values ('your_username', 'ROLE_USER');
+  ```
 
 1. Replace the H2 connection information in the application.yml file with MySQL connection
information:
-```
-spring:
-  datasource:
-        driverClassName: com.mysql.jdbc.Driver
-        url: jdbc:mysql://mysql_host:3306/metronrest
-        username: metron_rest_user
-        password: metron_rest_password
-        platform: mysql
-```
+  ```
+  spring:
+    datasource:
+          driverClassName: com.mysql.jdbc.Driver
+          url: jdbc:mysql://mysql_host:3306/metronrest
+          username: metron_rest_user
+          password: metron_rest_password
+          platform: mysql
+  ```
 
 1. Add a dependency for the MySQL JDBC connector in the metron-rest pom.xml:
-```
-<dependency>
-  <groupId>mysql</groupId>
-  <artifactId>mysql-connector-java</artifactId>
-  <version>${mysql.client.version}</version>
-</dependency>
-```
+  ```
+  <dependency>
+    <groupId>mysql</groupId>
+    <artifactId>mysql-connector-java</artifactId>
+    <version>${mysql.client.version}</version>
+  </dependency>
+  ```
 
 1. Follow the steps in the [Installation](#installation) section
 
@@ -95,25 +95,25 @@ Request and Response objects are JSON formatted.  The JSON schemas are
available
 | [ `GET /api/v1/hdfs/list`](#get-apiv1hdfslist)|
 | [ `GET /api/v1/kafka/topic`](#get-apiv1kafkatopic)|
 | [ `POST /api/v1/kafka/topic`](#post-apiv1kafkatopic)|
-| [ `GET /api/v1/kafka/topic/{name}`](#get-apiv1kafkatopic{name})|
-| [ `DELETE /api/v1/kafka/topic/{name}`](#delete-apiv1kafkatopic{name})|
-| [ `GET /api/v1/kafka/topic/{name}/sample`](#get-apiv1kafkatopic{name}sample)|
+| [ `GET /api/v1/kafka/topic/{name}`](#get-apiv1kafkatopicname)|
+| [ `DELETE /api/v1/kafka/topic/{name}`](#delete-apiv1kafkatopicname)|
+| [ `GET /api/v1/kafka/topic/{name}/sample`](#get-apiv1kafkatopicnamesample)|
 | [ `GET /api/v1/sensor/enrichment/config`](#get-apiv1sensorenrichmentconfig)|
 | [ `GET /api/v1/sensor/enrichment/config/list/available`](#get-apiv1sensorenrichmentconfiglistavailable)|
-| [ `DELETE /api/v1/sensor/enrichment/config/{name}`](#delete-apiv1sensorenrichmentconfig{name})|
-| [ `POST /api/v1/sensor/enrichment/config/{name}`](#post-apiv1sensorenrichmentconfig{name})|
-| [ `GET /api/v1/sensor/enrichment/config/{name}`](#get-apiv1sensorenrichmentconfig{name})|
+| [ `DELETE /api/v1/sensor/enrichment/config/{name}`](#delete-apiv1sensorenrichmentconfigname)|
+| [ `POST /api/v1/sensor/enrichment/config/{name}`](#post-apiv1sensorenrichmentconfigname)|
+| [ `GET /api/v1/sensor/enrichment/config/{name}`](#get-apiv1sensorenrichmentconfigname)|
 | [ `GET /api/v1/sensor/indexing/config`](#get-apiv1sensorindexingconfig)|
-| [ `DELETE /api/v1/sensor/indexing/config/{name}`](#delete-apiv1sensorindexingconfig{name})|
-| [ `POST /api/v1/sensor/indexing/config/{name}`](#post-apiv1sensorindexingconfig{name})|
-| [ `GET /api/v1/sensor/indexing/config/{name}`](#get-apiv1sensorindexingconfig{name})|
+| [ `DELETE /api/v1/sensor/indexing/config/{name}`](#delete-apiv1sensorindexingconfigname)|
+| [ `POST /api/v1/sensor/indexing/config/{name}`](#post-apiv1sensorindexingconfigname)|
+| [ `GET /api/v1/sensor/indexing/config/{name}`](#get-apiv1sensorindexingconfigname)|
 | [ `POST /api/v1/sensor/parser/config`](#post-apiv1sensorparserconfig)|
 | [ `GET /api/v1/sensor/parser/config`](#get-apiv1sensorparserconfig)|
 | [ `GET /api/v1/sensor/parser/config/list/available`](#get-apiv1sensorparserconfiglistavailable)|
 | [ `POST /api/v1/sensor/parser/config/parseMessage`](#post-apiv1sensorparserconfigparsemessage)|
 | [ `GET /api/v1/sensor/parser/config/reload/available`](#get-apiv1sensorparserconfigreloadavailable)|
-| [ `DELETE /api/v1/sensor/parser/config/{name}`](#delete-apiv1sensorparserconfig{name})|
-| [ `GET /api/v1/sensor/parser/config/{name}`](#get-apiv1sensorparserconfig{name})|
+| [ `DELETE /api/v1/sensor/parser/config/{name}`](#delete-apiv1sensorparserconfigname)|
+| [ `GET /api/v1/sensor/parser/config/{name}`](#get-apiv1sensorparserconfigname)|
 | [ `POST /api/v1/stellar/apply/transformations`](#post-apiv1stellarapplytransformations)|
 | [ `GET /api/v1/stellar/list`](#get-apiv1stellarlist)|
 | [ `GET /api/v1/stellar/list/functions`](#get-apiv1stellarlistfunctions)|
@@ -131,11 +131,11 @@ Request and Response objects are JSON formatted.  The JSON schemas are
available
 | [ `GET /api/v1/storm/indexing/deactivate`](#get-apiv1stormindexingdeactivate)|
 | [ `GET /api/v1/storm/indexing/start`](#get-apiv1stormindexingstart)|
 | [ `GET /api/v1/storm/indexing/stop`](#get-apiv1stormindexingstop)|
-| [ `GET /api/v1/storm/parser/activate/{name}`](#get-apiv1stormparseractivate{name})|
-| [ `GET /api/v1/storm/parser/deactivate/{name}`](#get-apiv1stormparserdeactivate{name})|
-| [ `GET /api/v1/storm/parser/start/{name}`](#get-apiv1stormparserstart{name})|
-| [ `GET /api/v1/storm/parser/stop/{name}`](#get-apiv1stormparserstop{name})|
-| [ `GET /api/v1/storm/{name}`](#get-apiv1storm{name})|
+| [ `GET /api/v1/storm/parser/activate/{name}`](#get-apiv1stormparseractivatename)|
+| [ `GET /api/v1/storm/parser/deactivate/{name}`](#get-apiv1stormparserdeactivatename)|
+| [ `GET /api/v1/storm/parser/start/{name}`](#get-apiv1stormparserstartname)|
+| [ `GET /api/v1/storm/parser/stop/{name}`](#get-apiv1stormparserstopname)|
+| [ `GET /api/v1/storm/{name}`](#get-apiv1stormname)|
 | [ `GET /api/v1/user`](#get-apiv1user)|
 
 ### `GET /api/v1/global/config`
@@ -171,7 +171,7 @@ Request and Response objects are JSON formatted.  The JSON schemas are
available
     * 200 - JSON results
 
 ### `POST /api/v1/hdfs`
-  * Description: Writes contents to an HDFS file.  Warning: this will overwite the contents
of a file if it already exists.
+  * Description: Writes contents to an HDFS file.  Warning: this will overwrite the contents
of a file if it already exists.
   * Input:
     * path - Path to HDFS file
     * contents - File contents
@@ -222,7 +222,7 @@ Request and Response objects are JSON formatted.  The JSON schemas are
available
     * 404 - Kafka topic is missing
 
 ### `DELETE /api/v1/kafka/topic/{name}`
-  * Description: Delets a Kafka topic
+  * Description: Deletes a Kafka topic
   * Input:
     * name - Kafka topic name
   * Returns:

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/metron-platform/metron-data-management/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-data-management/README.md b/metron-platform/metron-data-management/README.md
index 1bb7994..014f3ef 100644
--- a/metron-platform/metron-data-management/README.md
+++ b/metron-platform/metron-data-management/README.md
@@ -159,12 +159,12 @@ As an example, we will be providing a CSV list of top domains as an
enrichment a
 
 There are 2 property maps that work with full Stellar expressions, and 2 properties that
will work with Stellar predicates.
 
-| Property            | Description |
-|---------------------|-------------|
-| value_transform     | Transform fields defined in the "columns" mapping with Stellar transformations.
New keys introduced in the transform will be added to the key metadata. |
-| value_filter        | Allows additional filtering with Stellar predicates based on results
from the value transformations. In this example, records whose domain property is empty after
removing the TLD will be omitted. |
-| indicator_transform | Transform the indicator column independent of the value transformations.
You can refer to the original indicator value by using "indicator" as the variable name, as
shown in the example above. In addition, if you prefer to piggyback your transformations,
you can refer to the variable "domain", which will allow your indicator transforms to inherit
transformations done to this value during the value transformations. |
-| indicator_filter    | Allows additional filtering with Stellar predicates based on results
from the value transformations. In this example, records whose indicator value is empty after
removing the TLD will be omitted. |
+| Property            | Description
+|---------------------|---
+| value_transform     | Transform fields defined in the "columns" mapping with Stellar transformations.
New keys introduced in the transform will be added to the key metadata.
+| value_filter        | Allows additional filtering with Stellar predicates based on results
from the value transformations. In this example, records whose domain property is empty after
removing the TLD will be omitted.
+| indicator_transform | Transform the indicator column independent of the value transformations.
You can refer to the original indicator value by using "indicator" as the variable name, as
shown in the example above. In addition, if you prefer to piggyback your transformations,
you can refer to the variable "domain", which will allow your indicator transforms to inherit
transformations done to this value during the value transformations.
+| indicator_filter    | Allows additional filtering with Stellar predicates based on results
from the value transformations. In this example, records whose indicator value is empty after
removing the TLD will be omitted. 
 
 top-list.csv
 ```
@@ -175,10 +175,10 @@ top-list.csv
 
 Running a file import with the above data and extractor configuration would result in the
following 2 extracted data records:
 
-| Indicator | Type | Value |
-|-----------|------|-------|
+| Indicator | Type        | Value                                 |
+|-----------|-------------|---------------------------------------|
 | google    | top_domains | { "rank" : "1", "domain" : "google" } |
-| yahoo     | top_domains | { "rank" : "2", "domain" : "yahoo" } |
+| yahoo     | top_domains | { "rank" : "2", "domain" : "yahoo" }  |
 
 Similar to the parser framework, providing a Zookeeper quorum via the zk_quorum property
will enable Stellar to access properties that reside in the global config.
 Expanding on our example above, if the global config looks as follows:
@@ -196,15 +196,14 @@ And we expand our value_tranform:
        "a-new-prop" : "global_property"
     },
 ...
-
 ```
 
 The resulting value data would look like the following:
 
-| Indicator | Type | Value |
-|-----------|------|-------|
+| Indicator | Type        | Value                                                       
      |
+|-----------|-------------|--------------------------------------------------------------------|
 | google    | top_domains | { "rank" : "1", "domain" : "google", "a-new-prop" : "metron-ftw"
} |
-| yahoo     | top_domains | { "rank" : "2", "domain" : "yahoo", "a-new-prop" : "metron-ftw"
} |
+| yahoo     | top_domains | { "rank" : "2", "domain" : "yahoo", "a-new-prop" : "metron-ftw"
}  |
 
 ## Enrichment Config
 
@@ -303,19 +302,20 @@ each document to be considered as input to the Extractor.
 
 The parameters for the utility are as follows:
 
-| Short Code | Long Code           | Is Required? | Description                         
                                                                                         
                                                     |   |
-|------------|---------------------|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---|
-| -h         |                     | No           | Generate the help screen/set of options
                                                                                         
                                                  |   |
-| -q         | --quiet             | No           | Do not update progress
-| -e         | --extractor_config  | Yes          | JSON Document describing the extractor
for this input data source                                                               
                                                   |   |
-| -m         | --import_mode       | No           | The Import mode to use: LOCAL, MR.  Default:
LOCAL                                                                                    
                             |   |
-| -t         | --hbase_table       | Yes          | The HBase table to import into      
                                                                                         
                                                     |   |
-| -c         | --hbase_cf          | Yes          | The HBase table column family to import
into                                                                                     
                                                  |   |
-| -i         | --input             | Yes          | The input data location on local disk.
 If this is a file, then that file will be loaded.  If this is a directory, then the files
will be loaded recursively under that directory. |   |
-| -l         | --log4j             | No           | The log4j properties file to load   
                                                                                         
                                                     |   |
-| -n         | --enrichment_config | No           | The JSON document describing the enrichments
to configure.  Unlike other loaders, this is run first if specified.                     
                                             |   |
-| -p         | --threads           | No           | The number of threads to use when extracting
data.  The default is the number of cores.                                               
                                             |   |
-| -b         | --batchSize         | No           | The batch size to use for HBase puts
                                                                                         
                                                     |   |
+| Short Code | Long Code           | Is Required? | Description                         
                                                                                         
                                                     |
+|------------|---------------------|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| -h         |                     | No           | Generate the help screen/set of options
                                                                                         
                                                  |
+| -q         | --quiet             | No           | Do not update progress              
                                                                                         
                                                     |
+| -e         | --extractor_config  | Yes          | JSON Document describing the extractor
for this input data source                                                               
                                                   |
+| -m         | --import_mode       | No           | The Import mode to use: LOCAL, MR.  Default:
LOCAL                                                                                    
                                             |
+| -t         | --hbase_table       | Yes          | The HBase table to import into      
                                                                                         
                                                     |
+| -c         | --hbase_cf          | Yes          | The HBase table column family to import
into                                                                                     
                                                  |
+| -i         | --input             | Yes          | The input data location on local disk.
 If this is a file, then that file will be loaded.  If this is a directory, then the files
will be loaded recursively under that directory.  |
+| -l         | --log4j             | No           | The log4j properties file to load   
                                                                                         
                                                     |
+| -n         | --enrichment_config | No           | The JSON document describing the enrichments
to configure.  Unlike other loaders, this is run first if specified.                     
                                             |
+| -p         | --threads           | No           | The number of threads to use when extracting
data.  The default is the number of cores.                                               
                                             |
+| -b         | --batchSize         | No           | The batch size to use for HBase puts
                                                                                         
                                                     |
+
 ### GeoLite2 Loader
 
 The shell script `$METRON_HOME/bin/geo_enrichment_load.sh` will retrieve MaxMind GeoLite2
data and load data into HDFS, and update the configuration.
@@ -324,10 +324,10 @@ THIS SCRIPT WILL NOT UPDATE AMBARI'S GLOBAL.JSON, JUST THE ZK CONFIGS.
 CHANGES
 
 The parameters for the utility are as follows:
 
-| Short Code | Long Code           | Is Required? | Description                         
                                                                                         
                                                      |
-|------------|---------------------|--------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| -h         |                     | No           | Generate the help screen/set of options
                                                                                         
                                                   |
-| -g         | --geo_url           | No           | GeoIP URL - defaults to http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
-| -r         | --remote_dir        | No           | HDFS directory to land formatted GeoIP
file - defaults to /apps/metron/geo/\<epoch millis\>/
-| -t         | --tmp_dir           | No           | Directory for landing the temporary GeoIP
data - defaults to /tmp
-| -z         | --zk_quorum         | Yes          | Zookeeper Quorum URL (zk1:port,zk2:port,...)
+| Short Code | Long Code           | Is Required? | Description                         
                                                            |
+|------------|---------------------|--------------|--------------------------------------------------------------------------------------------------|
+| -h         |                     | No           | Generate the help screen/set of options
                                                         |
+| -g         | --geo_url           | No           | GeoIP URL - defaults to http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
|
+| -r         | --remote_dir        | No           | HDFS directory to land formatted GeoIP
file - defaults to /apps/metron/geo/\<epoch millis\>/     |
+| -t         | --tmp_dir           | No           | Directory for landing the temporary GeoIP
data - defaults to /tmp                                |
+| -z         | --zk_quorum         | Yes          | Zookeeper Quorum URL (zk1:port,zk2:port,...)
                                                    |

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/c793308c/site-book/bin/fix-md-dialect.py
----------------------------------------------------------------------
diff --git a/site-book/bin/fix-md-dialect.py b/site-book/bin/fix-md-dialect.py
index d594ca2..5e6db3e 100755
--- a/site-book/bin/fix-md-dialect.py
+++ b/site-book/bin/fix-md-dialect.py
@@ -59,7 +59,7 @@ import inspect
 import re
 
 # These are the characters excluded by Markdown from use in auto-generated anchor text for
Headings.
-EXCLUDED_CHARS_REGEX = r'[()[\]`"' + r"'" + r']'   # May add chars in future as needed
+EXCLUDED_CHARS_REGEX = r'[^\w\-]'   # all non-alphanumerics except "-" and "_".  Whitespace
are previously converted.
 
 def report_error(s) :
     print >>sys.stderr, "ERROR: " + s 
@@ -238,14 +238,16 @@ def rewrite_relative_links() :
         sharp = href.find("#")
         if (sharp >= 0) :
             named_anchor = href[sharp+1 : ]
-            scratch = labeltext.lower()
-            scratch = re.sub(r' ', "-", scratch)
-            scratch = re.sub(EXCLUDED_CHARS_REGEX, "", scratch)
+            trace('named_anchor = "' + named_anchor + '"')
+            trace('labeltext = "' + labeltext + '"')
+            scratch = labeltext.lower()                  # Github-MD forces all anchors to
lowercase
+            scratch = re.sub(r'[\s]', "-", scratch)      # convert whitespace to "-"
+            scratch = re.sub(EXCLUDED_CHARS_REGEX, "", scratch)  # strip non-alphanumerics
             if (scratch == named_anchor) :
                 trace("Found a rewritable case")
-                scratch = labeltext
-                scratch = re.sub(r' ', "_", scratch)
-                scratch = re.sub(EXCLUDED_CHARS_REGEX, "", scratch)
+                scratch = labeltext                      # Doxia-markdown doesn't change
case
+                scratch = re.sub(r'[\s]', "_", scratch)  # convert whitespace to "_"
+                scratch = re.sub(EXCLUDED_CHARS_REGEX, "", scratch)  # strip non-alphanumerics
                 href = re.sub("#" + named_anchor, "#" + scratch, href)
 
         trace("After anchor rewrite, href is: " + href)


Mime
View raw message