Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 2308F200C49 for ; Fri, 17 Mar 2017 15:51:53 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 211FA160B80; Fri, 17 Mar 2017 14:51:53 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D3698160B6D for ; Fri, 17 Mar 2017 15:51:50 +0100 (CET) Received: (qmail 35181 invoked by uid 500); 17 Mar 2017 14:51:50 -0000 Mailing-List: contact commits-help@metron.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.incubator.apache.org Delivered-To: mailing list commits@metron.incubator.apache.org Received: (qmail 35162 invoked by uid 99); 17 Mar 2017 14:51:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Mar 2017 14:51:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 70246C0DFC for ; Fri, 17 Mar 2017 14:51:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.869 X-Spam-Level: X-Spam-Status: No, score=-3.869 tagged_above=-999 required=6.31 tests=[KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_NEUTRAL=0.652] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id xID9oE4oys_l for ; Fri, 17 Mar 2017 14:51:44 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id D8AA85FCCA for ; Fri, 17 Mar 2017 14:51:41 +0000 (UTC) Received: (qmail 34479 invoked by uid 99); 17 Mar 2017 14:51:22 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Mar 2017 14:51:22 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 426E2DFF5E; Fri, 17 Mar 2017 14:51:22 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: cestella@apache.org To: commits@metron.incubator.apache.org Date: Fri, 17 Mar 2017 14:51:24 -0000 Message-Id: <2a2bdb22483c408a9cd0583287cf433c@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [03/15] incubator-metron git commit: METRON-766: Release 0.3.1 closes apache/incubator-metron#477 archived-at: Fri, 17 Mar 2017 14:51:53 -0000 http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/a055de44/current-book/metron-platform/metron-management/index.html ---------------------------------------------------------------------- diff --git a/current-book/metron-platform/metron-management/index.html b/current-book/metron-platform/metron-management/index.html new file mode 100644 index 0000000..c8d1d13 --- /dev/null +++ b/current-book/metron-platform/metron-management/index.html @@ -0,0 +1,1602 @@ + + + + + + + + + Metron – Stellar REPL Management Utilities + + + + + + + + + + + + + + + + + +
+ + + + + +
+
+
+ + + + + + +
+ +
+
+
+
+ + Built by Maven + +
+
+
+ + +
+ +

Stellar REPL Management Utilities

+

+

In order to augment the functionality of the Stellar REPL, a few management functions surrounding the management of the configurations and the management of Stellar transformations in the following areas have been added:

+ +
    + +
  • Stellar field transformations in the Parsers
    • + +
  • Stellar enrichments in the Enrichment topology
    • + +
  • Stellar threat triage rules
    • +
+

Additionally, some shell functions have been added to

+ +
    + +
  • provide the ability to refer to the Stellar expression used to create a variable
    • + +
  • print structured data in a way that is easier to view (i.e. tabular)
    • +
+

This functionality is exposed as a pack of Stellar functions in this project.

+
+

Function Details

+

The functions are split roughly into a few sections:

+ +
    + +
  • Shell functions - Functions surrounding interacting with the shell in either a nicer way or a more functional way.
    • + +
  • Grok Functions - Functions that allow you to evaluate grok expressions.
    • + +
  • File functions - Functions around interacting with local or HDFS files
    • + +
  • Configuration functions - Functions surrounding pulling and pushing configs from zookeeper
    • + +
  • Parser functions - Functions surrounding adding, viewing, and removing Parser functions.
    • + +
  • Enrichment functions - Functions surrounding adding, viewing and removing Stellar enrichments as well as managing batch size and index names for the enrichment topology configuration
    • + +
  • Threat Triage functions - Functions surrounding adding, viewing and removing threat triage functions.
    • +
+
+

Grok Functions

+ +
    + +
  • GROK_EVAL + +
      + +
    • Description: Evaluate a grok expression for a statement.
      • + +
    • Input: + +
        + +
      • grokExpression - The grok expression to evaluate
        • + +
      • data - Either a data message or a list of data messages to evaluate using the grokExpression
        • +
      • + +
    • Returns: The Map associated with the grok expression being evaluated on the list of messages.
      • +
    • + +
  • GROK_PREDICT + +
      + +
    • Description: Discover a grok statement for an input doc
      • + +
    • Input: + +
        + +
      • data - The data to discover a grok expression from
        • +
      • + +
    • Returns: A grok expression that should match the data.
      • +
    • +
+
+

File Functions

+ +
    + +
  • Local Files + +
      + +
    • LOCAL_LS + +
        + +
      • Description: Lists the contents of a directory.
        • + +
      • Input: + +
          + +
        • path - The path of the file
          • +
        • + +
      • Returns: The contents of the directory in tabular form sorted by last modification date.
        • +
      • + +
    • LOCAL_RM + +
        + +
      • Description: Removes the path
        • + +
      • Input: + +
          + +
        • path - The path of the file or directory.
          • + +
        • recursive - Recursively remove or not (optional and defaulted to false)
          • +
        • + +
      • Returns: boolean - true if successful, false otherwise
        • +
      • + +
    • LOCAL_READ + +
        + +
      • Description: Retrieves the contents as a string of a file.
        • + +
      • Input: + +
          + +
        • path - The path of the file
          • +
        • + +
      • Returns: The contents of the file and null otherwise.
        • +
      • + +
    • LOCAL_READ_LINES + +
        + +
      • Description: Retrieves the contents of a file as a list of strings.
        • + +
      • Input: + +
          + +
        • path - The path of the file
          • +
        • + +
      • Returns: A list of lines
        • +
      • + +
    • LOCAL_WRITE + +
        + +
      • Description: Writes the contents of a string to a local file
        • + +
      • Input: + +
          + +
        • content - The content to write out
          • + +
        • path - The path of the file
          • +
        • + +
      • Returns: true if the file was written and false otherwise.
        • +
      • +
    • + +
  • HDFS Files + +
      + +
    • HDFS_LS + +
        + +
      • Description: Lists the contents of a directory in HDFS.
        • + +
      • Input: + +
          + +
        • path - The path of the file
          • +
        • + +
      • Returns: The contents of the directory in tabular form sorted by last modification date.
        • +
      • + +
    • HDFS_RM + +
        + +
      • Description: Removes the path in HDFS.
        • + +
      • Input: + +
          + +
        • path - The path of the file or directory.
          • + +
        • recursive - Recursively remove or not (optional and defaulted to false)
          • +
        • + +
      • Returns: boolean - true if successful, false otherwise
        • +
      • + +
    • HDFS_READ + +
        + +
      • Description: Retrieves the contents as a string of a file in HDFS.
        • + +
      • Input: + +
          + +
        • path - The path of the file
          • +
        • + +
      • Returns: The contents of the file and null otherwise.
        • +
      • + +
    • HDFS_READ_LINES + +
        + +
      • Description: Retrieves the contents of a HDFS file as a list of strings.
        • + +
      • Input: + +
          + +
        • path - The path of the file
          • +
        • + +
      • Returns: A list of lines
        • +
      • + +
    • HDFS_WRITE + +
        + +
      • Description: Writes the contents of a string to a HDFS file
        • + +
      • Input: + +
          + +
        • content - The content to write out
          • + +
        • path - The path of the file
          • +
        • + +
      • Returns: true if the file was written and false otherwise.
        • +
      • +
    • +
+
+

Shell Functions

+ +
    + +
  • SHELL_EDIT + +
      + +
    • Description: Open an editor (optionally initialized with text) and return whatever is saved from the editor. The editor to use is pulled from EDITOR or VISUAL environment variable.
      • + +
    • Input: + +
        + +
      • string - (Optional) A string whose content is used to initialize the editor.
        • +
      • + +
    • Returns: The content that the editor saved after editor exit.
      • +
    • + +
  • SHELL_GET_EXPRESSION + +
      + +
    • Description: Get a stellar expression from a variable
      • + +
    • Input: + +
        + +
      • variable - variable name
        • +
      • + +
    • Returns: The stellar expression associated with the variable.
      • +
    • + +
  • SHELL_LIST_VARS + +
      + +
    • Description: Return the variables in a tabular form
      • + +
    • Input: + +
        + +
      • wrap : Length of string to wrap the columns
        • +
      • + +
    • Returns: A tabular representation of the variables.
      • +
    • + +
  • SHELL_MAP2TABLE + +
      + +
    • Description: Take a map and return a table
      • + +
    • Input: + +
        + +
      • map - Map
        • +
      • + +
    • Returns: The map in table form
      • +
    • + +
  • SHELL_VARS2MAP + +
      + +
    • Description: Take a set of variables and return a map
      • + +
    • Input: + +
        + +
      • variables* - variable names to use to create map
        • +
      • + +
    • Returns: A map associating the variable name with the stellar expression.
      • +
    • +
+
+

Configuration Functions

+ +
    + +
  • CONFIG_GET + +
      + +
    • Description: Retrieve a Metron configuration from zookeeper.
      • + +
    • Input: + +
        + +
      • type - One of ENRICHMENT, INDEXING, PARSER, GLOBAL, PROFILER
        • + +
      • sensor - Sensor to retrieve (required for enrichment and parser, not used for profiler and global)
        • + +
      • emptyIfNotPresent - If true, then return an empty, minimally viable config
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • + +
  • CONFIG_PUT + +
      + +
    • Description: Updates a Metron config to Zookeeper.
      • + +
    • Input: + +
        + +
      • type - One of ENRICHMENT, INDEXING, PARSER, GLOBAL, PROFILER
        • + +
      • config - The config (a string in JSON form) to update
        • + +
      • sensor - Sensor to retrieve (required for enrichment and parser, not used for profiler and global)
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • +
+
+

Parser Functions

+ +
    + +
  • PARSER_STELLAR_TRANSFORM_ADD + +
      + +
    • Description: Add stellar field transformation.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • stellarTransforms - A Map associating fields to stellar expressions
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • + +
  • PARSER_STELLAR_TRANSFORM_PRINT + +
      + +
    • Description: Retrieve stellar field transformations.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • +
      • + +
    • Returns: The String representation of the transformations
      • +
    • + +
  • PARSER_STELLAR_TRANSFORM_REMOVE + +
      + +
    • Description: Remove stellar field transformation.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • stellarTransforms - A list of stellar transforms to remove
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • +
+
+

Indexing Functions

+ +
    + +
  • INDEXING_SET_BATCH + +
      + +
    • Description: Set batch size
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • writer - The writer to update (e.g. elasticsearch, solr or hdfs)
        • + +
      • size - batch size (integer)
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • + +
  • INDEXING_SET_ENABLED + +
      + +
    • Description: Enable or disable an indexing writer for a sensor.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • writer - The writer to update (e.g. elasticsearch, solr or hdfs)
        • + +
      • enabled? - boolean indicating whether the writer is enabled. If omitted, then it will set enabled.
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • + +
  • INDEXING_SET_INDEX + +
      + +
    • Description: Set the index for the sensor
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • writer - The writer to update (e.g. elasticsearch, solr or hdfs)
        • + +
      • sensor - sensor name
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • +
+
+

Enrichment Functions

+ +
    + +
  • ENRICHMENT_STELLAR_TRANSFORM_ADD + +
      + +
    • Description: Add stellar field transformation.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • type - ENRICHMENT or THREAT_INTEL
        • + +
      • stellarTransforms - A Map associating fields to stellar expressions
        • + +
      • group - Group to add to (optional)
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • + +
  • ENRICHMENT_STELLAR_TRANSFORM_PRINT + +
      + +
    • Description: Retrieve stellar enrichment transformations.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • type - ENRICHMENT or THREAT_INTEL
        • +
      • + +
    • Returns: The String representation of the transformations
      • +
    • + +
  • ENRICHMENT_STELLAR_TRANSFORM_REMOVE + +
      + +
    • Description: Remove one or more stellar field transformations.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • type - ENRICHMENT or THREAT_INTEL
        • + +
      • stellarTransforms - A list of removals
        • + +
      • group - Group to remove from (optional)
        • +
      • + +
    • Returns: The String representation of the config in zookeeper
      • +
    • +
+
+

Threat Triage Functions

+ +
    + +
  • THREAT_TRIAGE_ADD + +
      + +
    • Description: Add a threat triage rule.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • stellarTransforms - A Map associating stellar rules to scores
        • + +
      • triageRules - Map (or list of Maps) representing a triage rule. It must contain ‘rule’ and ‘score’ keys, the stellar expression for the rule and triage score respectively. It may contain ‘name’ and ‘comment’, the name of the rule and comment associated with the rule respectively."
        • +
      • + +
    • Returns: The String representation of the threat triage rules
      • +
    • + +
  • THREAT_TRIAGE_PRINT + +
      + +
    • Description: Retrieve stellar enrichment transformations.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • +
      • + +
    • Returns: The String representation of the threat triage rules
      • +
    • + +
  • THREAT_TRIAGE_REMOVE + +
      + +
    • Description: Remove stellar threat triage rule(s).
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • rules - A list of stellar rules or rule names to remove
        • +
      • + +
    • Returns: The String representation of the enrichment config
      • +
    • + +
  • THREAT_TRIAGE_SET_AGGREGATOR + +
      + +
    • Description: Set the threat triage aggregator.
      • + +
    • Input: + +
        + +
      • sensorConfig - Sensor config to add transformation to.
        • + +
      • aggregator - Aggregator to use. One of MIN, MAX, MEAN, SUM, POSITIVE_MEAN
        • + +
      • aggregatorConfig - Optional config for aggregator
        • +
      • + +
    • Returns: The String representation of the enrichment config
      • +
    • +
+
+

Deployment Instructions

+

Deployment is as simple as dropping the jar created by this project into $METRON_HOME/lib and starting the Stellar shell via $METRON_HOME/bin/stellar

+
+

Examples

+

Included for description and education purposes are a couple example Stellar REPL transcripts with helpful comments to illustrate some common operations.

+
+

Iterate in finding a valid Grok pattern

+ +
+
+
Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+[Stellar]>>> # We are going to debug a squid grok statement with a bug in it
+[Stellar]>>> squid_grok_orig := '%{NUMBER:timestamp} %{SPACE:UNWANTED}  %{INT:elapsed} %{IP:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} 
+ - %{WORD:UNWANTED}/%{IP:ip_dst_addr} %{WORD:UNWANTED}/%{WORD:UNWANTED}'
+[Stellar]>>> # We have gone ot a couple of domains in squid:
+[Stellar]>>> #   1475022887.362    256 127.0.0.1 TCP_MISS/301 803 GET http://www.youtube.com/ - DIRECT/216.58.216.238 text/html
+[Stellar]>>> #   1475022915.731      1 127.0.0.1 NONE/400 3520 GET flimflammadeupdomain.com - NONE/- text/html
+[Stellar]>>> #   1475022938.661      0 127.0.0.1 NONE/400 3500 GET www.google.com - NONE/- text/html
+[Stellar]>>> # Note that flimflammadeupdomain.com and www.google.com did not resolve to IPs
+[Stellar]>>> # We can load up these messages from disk into a list of messages
+[Stellar]>>> messages := LOCAL_READ_LINES( '/var/log/squid/access.log')
+27687 [Thread-1] INFO  o.r.Reflections - Reflections took 26542 ms to scan 22 urls, producing 17906 keys and 121560 values 
+27837 [Thread-1] INFO  o.a.m.c.d.FunctionResolverSingleton - Found 97 Stellar Functions...
+Functions loaded, you may refer to functions now...
+[Stellar]>>> # and evaluate the messages against our grok statement
+[Stellar]>>> GROK_EVAL(squid_grok_orig, messages)
+╔══════════╤═════════╤═════════╤═════════╤════════════════╤═════════════╤═════════╤════════════════╤════════════════════════%
 50;╗
+║ action   │ bytes   │ code    │ elapsed │ ip_dst_addr    │ ip_src_addr │ method  │ timestamp      │ url                     ║
+╠══════════╪═════════╪═════════╪═════════╪════════════════╪═════════════╪═════════╪════════════════╪════════════════════════%
 50;╣
+║ TCP_MISS │ 803     │ 301     │ 256     │ 216.58.216.238 │ 127.0.0.1   │ GET     │ 1475022887.362 │ http://www.youtube.com/ ║
+╟──────────┼─────────┼─────────┼─────────┼────────────────┼─────────────┼─────────┼────────────────┼────────────────────────%
 00;╢
+║ MISSING  │ MISSING │ MISSING │ MISSING │ MISSING        │ MISSING     │ MISSING │ MISSING        │ MISSING                 ║
+╟──────────┼─────────┼─────────┼─────────┼────────────────┼─────────────┼─────────┼────────────────┼────────────────────────%
 00;╢
+║ MISSING  │ MISSING │ MISSING │ MISSING │ MISSING        │ MISSING     │ MISSING │ MISSING        │ MISSING                 ║
+╚══════════╧═════════╧═════════╧═════════╧════════════════╧═════════════╧═════════╧════════════════╧════════════════════════%
 50;╝
+
+[Stellar]>>> # Uh oh, looks like the messages without destination IPs do not parse
+[Stellar]>>> # We can start peeling off groups from the end of the message until things parse
+[Stellar]>>> squid_grok := '%{NUMBER:timestamp} %{SPACE:UNWANTED}  %{INT:elapsed} %{IP:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} - %{ 
+WORD:UNWANTED}/%{IP:ip_dst_addr}'
+[Stellar]>>> GROK_EVAL(squid_grok, messages)
+╔══════════╤═════════╤═════════╤═════════╤════════════════╤═════════════╤═════════╤════════════════╤════════════════════════%
 50;╗
+║ action   │ bytes   │ code    │ elapsed │ ip_dst_addr    │ ip_src_addr │ method  │ timestamp      │ url                     ║
+╠══════════╪═════════╪═════════╪═════════╪════════════════╪═════════════╪═════════╪════════════════╪════════════════════════%
 50;╣
+║ TCP_MISS │ 803     │ 301     │ 256     │ 216.58.216.238 │ 127.0.0.1   │ GET     │ 1475022887.362 │ http://www.youtube.com/ ║
+╟──────────┼─────────┼─────────┼─────────┼────────────────┼─────────────┼─────────┼────────────────┼────────────────────────%
 00;╢
+║ MISSING  │ MISSING │ MISSING │ MISSING │ MISSING        │ MISSING     │ MISSING │ MISSING        │ MISSING                 ║
+╟──────────┼─────────┼─────────┼─────────┼────────────────┼─────────────┼─────────┼────────────────┼────────────────────────%
 00;╢
+║ MISSING  │ MISSING │ MISSING │ MISSING │ MISSING        │ MISSING     │ MISSING │ MISSING        │ MISSING                 ║
+╚══════════╧═════════╧═════════╧═════════╧════════════════╧═════════════╧═════════╧════════════════╧════════════════════════%
 50;╝
+
+[Stellar]>>> # Still looks like it is having issues...
+[Stellar]>>> squid_grok := '%{NUMBER:timestamp} %{SPACE:UNWANTED}  %{INT:elapsed} %{IP:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} - %{ 
+WORD:UNWANTED}/%{IP:ip_dst_addr}'
+[Stellar]>>> GROK_EVAL(squid_grok, messages)
+╔══════════╤═════════╤═════════╤═════════╤════════════════╤═════════════╤═════════╤════════════════╤════════════════════════%
 50;╗
+║ action   │ bytes   │ code    │ elapsed │ ip_dst_addr    │ ip_src_addr │ method  │ timestamp      │ url                     ║
+╠══════════╪═════════╪═════════╪═════════╪════════════════╪═════════════╪═════════╪════════════════╪════════════════════════%
 50;╣
+║ TCP_MISS │ 803     │ 301     │ 256     │ 216.58.216.238 │ 127.0.0.1   │ GET     │ 1475022887.362 │ http://www.youtube.com/ ║
+╟──────────┼─────────┼─────────┼─────────┼────────────────┼─────────────┼─────────┼────────────────┼────────────────────────%
 00;╢
+║ MISSING  │ MISSING │ MISSING │ MISSING │ MISSING        │ MISSING     │ MISSING │ MISSING        │ MISSING                 ║
+╟──────────┼─────────┼─────────┼─────────┼────────────────┼─────────────┼─────────┼────────────────┼────────────────────────%
 00;╢
+║ MISSING  │ MISSING │ MISSING │ MISSING │ MISSING        │ MISSING     │ MISSING │ MISSING        │ MISSING                 ║
+╚══════════╧═════════╧═════════╧═════════╧════════════════╧═════════════╧═════════╧════════════════╧════════════════════════%
 50;╝
+
+[Stellar]>>> # Still looks wrong.  Hmm, I bet it is due to that dst_addr not being there; we can make it optional
+[Stellar]>>> squid_grok := '%{NUMBER:timestamp} %{SPACE:UNWANTED}  %{INT:elapsed} %{IP:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} - %{ 
+WORD:UNWANTED}/(%{IP:ip_dst_addr})?'
+[Stellar]>>> GROK_EVAL(squid_grok, messages)
+╔══════════╤═══════╤══════╤═════════╤════════════════╤═════════════╤════════╤════════════════╤══════════════════════════╗
+║ action   │ bytes │ code │ elapsed │ ip_dst_addr    │ ip_src_addr │ method │ timestamp      │ url                      ║
+╠══════════╪═══════╪══════╪═════════╪════════════════╪═════════════╪════════╪════════════════╪══════════════════════════╣
+║ TCP_MISS │ 803   │ 301  │ 256     │ 216.58.216.238 │ 127.0.0.1   │ GET    │ 1475022887.362 │ http://www.youtube.com/  ║
+╟──────────┼───────┼──────┼─────────┼────────────────┼─────────────┼────────┼────────────────┼──────────────────────────╢
+║ NONE     │ 3520  │ 400  │ 1       │ null           │ 127.0.0.1   │ GET    │ 1475022915.731 │ flimflammadeupdomain.com ║
+╟──────────┼───────┼──────┼─────────┼────────────────┼─────────────┼────────┼────────────────┼──────────────────────────╢
+║ NONE     │ 3500  │ 400  │ 0       │ null           │ 127.0.0.1   │ GET    │ 1475022938.661 │ www.google.com           ║
+╚══════════╧═══════╧══════╧═════════╧════════════════╧═════════════╧════════╧════════════════╧══════════════════════════╝
+
+[Stellar]>>> # Ahh, that is much better.
+[Stellar]>>> 
+[Stellar]>>> 
+
+
+

Manage Stellar Field Transformations

+ +
+
+
964  [main] INFO  o.a.c.f.i.CuratorFrameworkImpl - Starting
+1025 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
+Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
+[Stellar]>>> # First we get the squid parser config from zookeeper
+[Stellar]>>> squid_parser_config := CONFIG_GET('PARSER', 'squid')
+29089 [Thread-1] INFO  o.r.Reflections - Reflections took 26765 ms to scan 22 urls, producing 17898 keys and 121518 values 
+29177 [Thread-1] INFO  o.a.m.c.d.FunctionResolverSingleton - Found 83 Stellar Functions...
+Functions loaded, you may refer to functions now...
+[Stellar]>>> # See what kind of transformations it already has
+[Stellar]>>> PARSER_STELLAR_TRANSFORM_PRINT(squid_parser_config)
+╔═══════════════════════════╤═════════════════════════════════════════╗
+║ Field                     │ Transformation                          ║
+╠═══════════════════════════╪═════════════════════════════════════════╣
+║ full_hostname             │ URL_TO_HOST(url)                        ║
+╟───────────────────────────┼─────────────────────────────────────────╢
+║ domain_without_subdomains │ DOMAIN_REMOVE_SUBDOMAINS(full_hostname) ║
+╚═══════════════════════════╧═════════════════════════════════════════╝
+
+[Stellar]>>> #Just to make sure it looks right, we can view the JSON
+[Stellar]>>> squid_parser_config
+{
+  "parserClassName": "org.apache.metron.parsers.GrokParser",
+  "sensorTopic": "squid",
+  "parserConfig": {
+    "grokPath": "/patterns/squid",
+    "patternLabel": "SQUID_DELIMITED",
+    "timestampField": "timestamp"
+  },
+  "fieldTransformations" : [
+    {
+      "transformation" : "STELLAR"
+    ,"output" : [ "full_hostname", "domain_without_subdomains" ]
+    ,"config" : {
+      "full_hostname" : "URL_TO_HOST(url)"
+      ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
+                }
+    }
+                           ]
+}
+
+[Stellar]>>> # Add another transformation in there
+[Stellar]>>> domain_without_subdomains := 'cnn.com'
+[Stellar]>>> upper_domain := TO_UPPER(domain_without_subdomains)
+[Stellar]>>> # Now we can look at our variables and see what expressions created them 
+[Stellar]>>> # NOTE: the 40 is the max char for a word
+[Stellar]>>> SHELL_LIST_VARS( 40 )
+╔═══════════════════════════╤═══════════════════════════════════════════╤═════════════════════════════════════╗
+║ VARIABLE                  │ VALUE                                     │ EXPRESSION                          ║
+╠═══════════════════════════╪═══════════════════════════════════════════╪═════════════════════════════════════╣
+║ squid_parser_config       │ {                                         │ CONFIG_GET('PARSER', 'squid')       ║
+║                           │   "parserClassName":                      │                                     ║
+║                           │ "org.apache.metron.parsers.GrokParser",   │                                     ║
+║                           │                                           │                                     ║
+║                           │ "sensorTopic": "squid",                   │                                     ║
+║                           │                                           │                                     ║
+║                           │ "parserConfig": {                         │                                     ║
+║                           │     "grokPath":                           │                                     ║
+║                           │ "/patterns/squid",                        │                                     ║
+║                           │     "patternLabel":                       │                                     ║
+║                           │ "SQUID_DELIMITED",                        │                                     ║
+║                           │     "timestampField":                     │                                     ║
+║                           │ "timestamp"                               │                                     ║
+║                           │   },                                      │                                     ║
+║                           │                                           │                                     ║
+║                           │ "fieldTransformations" : [                │                                     ║
+║                           │     {                                     │                                     ║
+║                           │                                           │                                     ║
+║                           │ "transformation" : "STELLAR"              │                                     ║
+║                           │                                           │                                     ║
+║                           │ ,"output" : [ "full_hostname",            │                                     ║
+║                           │ "domain_without_subdomains" ]             │                                     ║
+║                           │                                           │                                     ║
+║                           │ ,"config" : {                             │                                     ║
+║                           │       "full_hostname" :                   │                                     ║
+║                           │ "URL_TO_HOST(url)"                        │                                     ║
+║                           │                                           │                                     ║
+║                           │ ,"domain_without_subdomains" :            │                                     ║
+║                           │ "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)" │                                     ║
+║                           │                                           │                                     ║
+║                           │  }                                        │                                     ║
+║                           │     }                                     │                                     ║
+║                           │                            ]              │                                     ║
+║                           │ }                                         │                                     ║
+╟───────────────────────────┼───────────────────────────────────────────┼─────────────────────────────────────╢
+║ domain_without_subdomains │ cnn.com                                   │ 'cnn.com'                           ║
+╟───────────────────────────┼───────────────────────────────────────────┼─────────────────────────────────────╢
+║ upper_domain              │ CNN.COM                                   │ TO_UPPER(domain_without_subdomains) ║
+╚═══════════════════════════╧═══════════════════════════════════════════╧═════════════════════════════════════╝
+
+[Stellar]>>> # We can add upper_domain as a transformation to the parser now by
+[Stellar]>>> ?PARSER_STELLAR_TRANSFORM_ADD
+PARSER_STELLAR_TRANSFORM_ADD
+Description: Add stellar field transformation.                           
+
+Arguments:
+    sensorConfig - Sensor config to add transformation to.      
+    stellarTransforms - A Map associating fields to stellar expressions
+
+Returns: The String representation of the config in zookeeper        
+[Stellar]>>> # We will use the SHELL_VARS2MAP to construct our map associating the field name with the expression
+[Stellar]>>> ?SHELL_VARS2MAP
+SHELL_VARS2MAP
+Description: Take a set of variables and return a map                    
+
+Arguments:
+    variables* - variable names to use to create map            
+
+Returns: A map associating the variable name with the stellar expression.
+[Stellar]>>> squid_parser_config_new := PARSER_STELLAR_TRANSFORM_ADD( squid_parser_config, SHELL_VARS2MAP('upper_domain') )
+[Stellar]>>> #Now we can make sure that we have the transformation added
+[Stellar]>>> PARSER_STELLAR_TRANSFORM_PRINT(squid_parser_config_new)
+╔═══════════════════════════╤═════════════════════════════════════════╗
+║ Field                     │ Transformation                          ║
+╠═══════════════════════════╪═════════════════════════════════════════╣
+║ full_hostname             │ URL_TO_HOST(url)                        ║
+╟───────────────────────────┼─────────────────────────────────────────╢
+║ domain_without_subdomains │ DOMAIN_REMOVE_SUBDOMAINS(full_hostname) ║
+╟───────────────────────────┼─────────────────────────────────────────╢
+║ upper_domain              │ TO_UPPER(domain_without_subdomains)     ║
+╚═══════════════════════════╧═════════════════════════════════════════╝
+
+[Stellar]>>> #And finally, we push the configs back to zookeeper
+[Stellar]>>> CONFIG_PUT('PARSER', squid_parser_config_new, 'squid')
+[Stellar]>>> #Now we can make sure that we have the transformation added into zookeeper
+[Stellar]>>> PARSER_STELLAR_TRANSFORM_PRINT(CONFIG_GET('PARSER', 'squid'))
+╔═══════════════════════════╤═════════════════════════════════════════╗
+║ Field                     │ Transformation                          ║
+╠═══════════════════════════╪═════════════════════════════════════════╣
+║ full_hostname             │ URL_TO_HOST(url)                        ║
+╟───────────────────────────┼─────────────────────────────────────────╢
+║ domain_without_subdomains │ DOMAIN_REMOVE_SUBDOMAINS(full_hostname) ║
+╟───────────────────────────┼─────────────────────────────────────────╢
+║ upper_domain              │ TO_UPPER(domain_without_subdomains)     ║
+╚═══════════════════════════╧═════════════════════════════════════════╝
+
+[Stellar]>>> #Now that we have added it, we can change our mind and remove it
+[Stellar]>>> ?PARSER_STELLAR_TRANSFORM_REMOVE
+PARSER_STELLAR_TRANSFORM_REMOVE
+Description: Remove stellar field transformation.                        
+
+Arguments:
+    sensorConfig - Sensor config to add transformation to.      
+    stellarTransforms - A list of stellar transforms to remove  
+
+Returns: The String representation of the config in zookeeper        
+[Stellar]>>> squid_parser_config_new := PARSER_STELLAR_TRANSFORM_REMOVE( CONFIG_GET('PARSER', 'squid'), [ 'upper_domain' ] )
+[Stellar]>>> PARSER_STELLAR_TRANSFORM_PRINT(squid_parser_config_new)
+╔═══════════════════════════╤═════════════════════════════════════════╗
+║ Field                     │ Transformation                          ║
+╠═══════════════════════════╪═════════════════════════════════════════╣
+║ full_hostname             │ URL_TO_HOST(url)                        ║
+╟───────────────────────────┼─────────────────────────────────────────╢
+║ domain_without_subdomains │ DOMAIN_REMOVE_SUBDOMAINS(full_hostname) ║
+╚═══════════════════════════╧═════════════════════════════════════════╝
+
+[Stellar]>>> #We can now push up the config to zookeeper
+[Stellar]>>> CONFIG_PUT('PARSER', squid_parser_config_new, 'squid')
+[Stellar]>>> #It should be just as we started the exercise
+[Stellar]>>> CONFIG_GET('PARSER', 'squid')
+{
+  "parserClassName" : "org.apache.metron.parsers.GrokParser",
+  "sensorTopic" : "squid",
+  "parserConfig" : {
+    "grokPath" : "/patterns/squid",
+    "patternLabel" : "SQUID_DELIMITED",
+    "timestampField" : "timestamp"
+  },
+  "fieldTransformations" : [ {
+    "input" : [ ],
+    "output" : [ "full_hostname", "domain_without_subdomains" ],
+    "transformation" : "STELLAR",
+    "config" : {
+      "full_hostname" : "URL_TO_HOST(url)",
+      "domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
+    }
+  } ]
+}
+[Stellar]>>> #And quit the REPL
+[Stellar]>>> quit
+
+
+

Manage Stellar Enrichments

+ +
+
+
1010 [main] INFO  o.a.c.f.i.CuratorFrameworkImpl - Starting
+1077 [main-EventThread] INFO  o.a.c.f.s.ConnectionStateManager - State change: CONNECTED
+Stellar, Go!
+Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
+{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH}
+[Stellar]>>> # First we get the squid enrichment config from zookeeper.
+[Stellar]>>> # If it is not there, which it is not by default, a suitable default
+[Stellar]>>> # config will be specified.
+[Stellar]>>> squid_enrichment_config := CONFIG_GET('ENRICHMENT', 'squid')
+26307 [Thread-1] INFO  o.r.Reflections - Reflections took 24845 ms to scan 22 urls, producing 17898 keys and 121520 values 
+26389 [Thread-1] INFO  o.a.m.c.d.FunctionResolverSingleton - Found 84 Stellar Functions...
+Functions loaded, you may refer to functions now...
+[Stellar]>>> # Just to make sure it looks right, we can view the JSON
+[Stellar]>>> squid_enrichment_config
+{
+  "enrichment" : {
+    "fieldMap" : { },
+    "fieldToTypeMap" : { },
+    "config" : { }
+  },
+  "threatIntel" : {
+    "fieldMap" : { },
+    "fieldToTypeMap" : { },
+    "config" : { },
+    "triageConfig" : {
+      "riskLevelRules" : [ ],
+      "aggregator" : "MAX",
+      "aggregationConfig" : { }
+    }
+  },
+  "configuration" : { }
+}
+[Stellar]>>> # Now that we have a config, we can add an enrichment to the Stellar adapter
+[Stellar]>>> # We should make sure that the current enrichment does not have any already
+[Stellar]>>> ?ENRICHMENT_STELLAR_TRANSFORM_PRINT
+ENRICHMENT_STELLAR_TRANSFORM_PRINT
+Description: Retrieve stellar enrichment transformations.                
+
+Arguments:
+    sensorConfig - Sensor config to add transformation to.      
+    type - ENRICHMENT or THREAT_INTEL                           
+
+Returns: The String representation of the transformations            
+[Stellar]>>> # Since there are two places we can add enrichments, we should check both
+[Stellar]>>> ENRICHMENT_STELLAR_TRANSFORM_PRINT(squid_enrichment_config, 'ENRICHMENT')
+╔═══════╤═══════╤════════════════╗
+║ Group │ Field │ Transformation ║
+╠═══════╧═══════╧════════════════╣
+║ (empty)                        ║
+╚════════════════════════════════╝
+
+[Stellar]>>> ENRICHMENT_STELLAR_TRANSFORM_PRINT(squid_enrichment_config, 'THREAT_INTEL')
+╔═══════╤═══════╤════════════════╗
+║ Group │ Field │ Transformation ║
+╠═══════╧═══════╧════════════════╣
+║ (empty)                        ║
+╚════════════════════════════════╝
+
+[Stellar]>>> # For my enrichment, I want to add a field indicating that the src address is local or not
+[Stellar]>>> # I will define local as part of '192.168.0.0/21'
+[Stellar]>>> # We must be careful about the ip_src_addr, what if it is malformed?
+[Stellar]>>> ip_src_addr := '200.20.10.1'
+[Stellar]>>> IN_SUBNET( ip_src_addr, '192.168.0.0/21')
+false
+[Stellar]>>> # Just as we expected.  Now we can try a local address
+[Stellar]>>> ip_src_addr := '192.168.0.1'
+[Stellar]>>> IN_SUBNET( ip_src_addr, '192.168.0.0/21')
+true
+[Stellar]>>> # Just as we expected.  Now we can try some malformed ones
+[Stellar]>>> ip_src_addr := NULL
+[Stellar]>>> IN_SUBNET( ip_src_addr, '192.168.0.0/21')
+false
+[Stellar]>>> # So far, so good
+[Stellar]>>> ip_src_addr := 'foo bar'
+[Stellar]>>> IN_SUBNET( ip_src_addr, '192.168.0.0/21')
+[!] Unable to execute: Could not parse [foo bar]
+[Stellar]>>> # uh oh, that was terrible, we will have to adjust and be a bit more defensive
+[Stellar]>>> IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
+false
+[Stellar]>>> ip_src_addr := NULL
+[Stellar]>>> IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
+false
+[Stellar]>>> ip_src_addr := '192.168.0.1'
+[Stellar]>>> IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
+true
+[Stellar]>>> ip_src_addr := '200.20.10.1'
+[Stellar]>>> IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
+false
+[Stellar]>>> # I think we are ready, we will call it is_local
+[Stellar]>>> is_local := IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21')
+[Stellar]>>> # Now we can add the function to the ENRICHMENT phase of the enrichment topology
+[Stellar]>>> ?ENRICHMENT_STELLAR_TRANSFORM_ADD
+ENRICHMENT_STELLAR_TRANSFORM_ADD
+Description: Add stellar field transformation.                           
+
+Arguments:
+    sensorConfig - Sensor config to add transformation to.      
+    type - ENRICHMENT or THREAT_INTEL                           
+    stellarTransforms - A Map associating fields to stellar expressions
+    group - Group to add to (optional)                          
+
+Returns: The String representation of the config in zookeeper        
+[Stellar]>>> squid_enrichment_config_new := ENRICHMENT_STELLAR_TRANSFORM_ADD( squid_enrichment_config, 'ENRICHMENT', SHELL_VARS2MAP( 'is_local' ) )
+[Stellar]>>> # Make sure that it is really there
+[Stellar]>>> ENRICHMENT_STELLAR_TRANSFORM_PRINT(squid_enrichment_config_new, 'ENRICHMENT')
+╔═══════════╤══════════╤════════════════════════════════════════════════════════════════════════════════╗
+║ Group     │ Field    │ Transformation                                                                 ║
+╠═══════════╪══════════╪════════════════════════════════════════════════════════════════════════════════╣
+║ (default) │ is_local │ IN_SUBNET( if IS_IP(ip_src_addr) then ip_src_addr else NULL, '192.168.0.0/21') ║
+╚═══════════╧══════════╧════════════════════════════════════════════════════════════════════════════════╝
+
+[Stellar]>>> # and not in threat intel
+[Stellar]>>> ENRICHMENT_STELLAR_TRANSFORM_PRINT(squid_enrichment_config_new, 'THREAT_INTEL')
+╔═══════╤═══════╤════════════════╗
+║ Group