metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ceste...@apache.org
Subject incubator-metron git commit: METRON-637: Add a STATS_BIN function to Stellar. closes apache/incubator-metron#401
Date Wed, 11 Jan 2017 14:11:34 GMT
Repository: incubator-metron
Updated Branches:
  refs/heads/master e7fcc755c -> 09cb50288


METRON-637: Add a STATS_BIN function to Stellar. closes apache/incubator-metron#401


Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/09cb5028
Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/09cb5028
Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/09cb5028

Branch: refs/heads/master
Commit: 09cb50288f3044e30e6674fca03ef2d05483cff8
Parents: e7fcc75
Author: cstella <cestella@gmail.com>
Authored: Wed Jan 11 09:11:25 2017 -0500
Committer: cstella <cestella@gmail.com>
Committed: Wed Jan 11 09:11:25 2017 -0500

----------------------------------------------------------------------
 metron-analytics/metron-statistics/README.md    |  21 ++-
 .../apache/metron/statistics/MathFunctions.java |  50 +++++-
 .../statistics/StellarStatisticsFunctions.java  |  56 +++++++
 .../metron/statistics/MathFunctionsTest.java    |  15 ++
 .../StatisticalBinningPerformanceDriver.java    |  76 +++++++++
 .../StellarStatisticsFunctionsTest.java         |  53 +++++-
 metron-platform/metron-common/README.md         | 161 ++++++++++---------
 7 files changed, 346 insertions(+), 86 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/README.md b/metron-analytics/metron-statistics/README.md
index edc68e1..346179b 100644
--- a/metron-analytics/metron-statistics/README.md
+++ b/metron-analytics/metron-statistics/README.md
@@ -11,11 +11,19 @@ functions can be used from everywhere where Stellar is used.
 ## Stellar Functions
 
 ### Mathematical Functions
+
 #### `ABS`
 * Description: Returns the absolute value of a number.
 * Input:
   * number - The number to take the absolute value of
 * Returns: The absolute value of the number passed in.
+*
+#### `BIN`
+* Description: Computes the bin that the value is in given a set of bounds.
+* Input:
+  * value - The value to bin
+  * bounds - A list of value bounds (excluding min and max) in sorted order.
+* Returns: Which bin N the value falls in such that bound(N-1) < value <= bound(N).
 No min and max bounds are provided, so values smaller than the 0'th bound go in the 0'th
bin, and values greater than the last bound go in the M'th bin.
 
 
 ### Distributional Statistics
@@ -27,6 +35,14 @@ functions can be used from everywhere where Stellar is used.
     * value+ - One or more numbers to add
   * Returns: A Stellar statistics object
 
+#### `STATS_BIN`
+  * Description: Computes the bin that the value is in based on the statistical distribution.

+  * Input:
+    * stats - The Stellar statistics object
+    * value - The value to bin
+    * bounds? - A list of percentile bin bounds (excluding min and max) or a string representing
a known and common set of bins.  For convenience, we have provided QUARTILE, QUINTILE, and
DECILE which you can pass in as a string arg. If this argument is omitted, then we assume
a Quartile bin split. 
+  * Returns: "Which bin N the value falls in such that bound(N-1) < value <= bound(N).
No min and max bounds are provided, so values smaller than the 0'th bound go in the 0'th bin,
and values greater than the last bound go in the M'th bin.
+ 
 #### `STATS_COUNT`
   * Description: Calculates the count of the values accumulated (or in the window if a window
is used).
   * Input:
@@ -129,8 +145,9 @@ functions can be used from everywhere where Stellar is used.
   * Input:
     * stats - The Stellar statistics object
   * Returns: The variance of the values in the window or NaN if the statistics object is
null.
-  
-  ### Statistical Outlier Detection
+
+
+### Statistical Outlier Detection
   
 #### `OUTLIER_MAD_STATE_MERGE`
   * Description: Update the statistical state required to compute the Median Absolute Deviation.

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java
b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java
index 548a611..9364095 100644
--- a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java
+++ b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java
@@ -19,12 +19,13 @@
  */
 package org.apache.metron.statistics;
 
-import org.apache.metron.common.dsl.Context;
-import org.apache.metron.common.dsl.ParseException;
-import org.apache.metron.common.dsl.Stellar;
-import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.*;
 
+import java.util.ArrayList;
 import java.util.List;
+import java.util.function.Function;
+
+import static org.apache.metron.common.utils.ConversionUtils.convert;
 
 public class MathFunctions {
 
@@ -59,4 +60,45 @@ public class MathFunctions {
       return true;
     }
   }
+
+  /**
+   * Calculates the statistical bin that a value falls in.
+   */
+  @Stellar(name = "BIN"
+          , description = "Computes the bin that the value is in given a set of bounds."
+          , params = {
+           "value - The value to bin"
+          , "bounds - A list of value bounds (excluding min and max) in sorted order."
+                    }
+          ,returns = "Which bin N the value falls in such that bound(N-1) < value <=
bound(N). " +
+          "No min and max bounds are provided, so values smaller than the 0'th bound go in
the 0'th bin, " +
+          "and values greater than the last bound go in the M'th bin."
+  )
+  public static class Bin extends BaseStellarFunction {
+
+    public static int getBin(double value, int numBins, Function<Integer, Double> boundFunc)
{
+      double lastBound = Double.NEGATIVE_INFINITY;
+      for(int bin = 0; bin < numBins;++bin) {
+        double bound = boundFunc.apply(bin);
+        if(lastBound > bound ) {
+          throw new IllegalStateException("Your bins must be non-decreasing");
+        }
+        if(value <= bound) {
+          return bin;
+        }
+        lastBound = bound;
+      }
+      return numBins;
+    }
+
+    @Override
+    public Object apply(List<Object> args) {
+      Double value = convert(args.get(0), Double.class);
+      final List<Number> bins = args.size() > 1?convert(args.get(1), List.class):null;
+      if ( value == null || bins == null || bins.size() == 0) {
+        return -1;
+      }
+      return getBin(value, bins.size(), bin -> bins.get(bin).doubleValue());
+    }
+  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
index 2ea8ee1..2223b47 100644
--- a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
+++ b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
@@ -20,9 +20,12 @@
 
 package org.apache.metron.statistics;
 
+import com.google.common.collect.ImmutableList;
 import org.apache.metron.common.dsl.BaseStellarFunction;
 import org.apache.metron.common.dsl.Stellar;
+import org.apache.metron.common.utils.ConversionUtils;
 
+import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
 
@@ -425,4 +428,57 @@ public class StellarStatisticsFunctions {
       return result;
     }
   }
+
+  /**
+   * Calculates the statistical bin that a value falls in.
+   */
+  @Stellar(namespace = "STATS", name = "BIN"
+          , description = "Computes the bin that the value is in based on the statistical
distribution."
+          , params = {
+          "stats - The Stellar statistics object"
+          , "value - The value to bin"
+          , "bounds? - A list of percentile bin bounds (excluding min and max) or a string
representing a known and common set of bins.  " +
+          "For convenience, we have provided QUARTILE, QUINTILE, and DECILE which you can
pass in as a string arg." +
+          " If this argument is omitted, then we assume a Quartile bin split."
+                    }
+          ,returns = "Which bin N the value falls in such that bound(N-1) < value <=
bound(N). " +
+          "No min and max bounds are provided, so values smaller than the 0'th bound go in
the 0'th bin, " +
+          "and values greater than the last bound go in the M'th bin."
+  )
+  public static class StatsBin extends BaseStellarFunction {
+    public enum BinSplits {
+      QUARTILE(ImmutableList.of(25.0, 50.0, 75.0)),
+      QUINTILE(ImmutableList.of(20.0, 40.0, 60.0, 80.0)),
+      DECILE(ImmutableList.of(10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0))
+      ;
+      public final List<Number> split;
+      BinSplits(List<Number> split) {
+        this.split = split;
+      }
+
+      public static List<Number> getSplit(Object o) {
+        if(o instanceof String) {
+          return BinSplits.valueOf((String)o).split;
+        }
+        else if(o instanceof List) {
+          return ConversionUtils.convert(o, List.class);
+        }
+        throw new IllegalStateException("The split you tried to pass is not a valid split:
" + o.toString());
+      }
+    }
+
+
+    @Override
+    public Object apply(List<Object> args) {
+      StatisticsProvider stats = convert(args.get(0), StatisticsProvider.class);
+      Double value = convert(args.get(1), Double.class);
+      final List<Number> bins = args.size() > 2?BinSplits.getSplit(args.get(2)):BinSplits.QUARTILE.split;
+
+      if (stats == null || value == null || bins.size() == 0) {
+        return -1;
+      }
+      return MathFunctions.Bin.getBin(value, bins.size(), bin -> stats.getPercentile(bins.get(bin).doubleValue()));
+    }
+  }
+
 }

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java
b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java
index 91e4de5..c615966 100644
--- a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java
+++ b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java
@@ -19,6 +19,7 @@
  */
 package org.apache.metron.statistics;
 
+import com.google.common.collect.ImmutableList;
 import com.google.common.collect.ImmutableMap;
 import org.apache.metron.common.dsl.Context;
 import org.apache.metron.common.dsl.StellarFunctions;
@@ -43,4 +44,18 @@ public class MathFunctionsTest {
     Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", 10.5)), 10.5,
1e-7);
     Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", -10.5)), 10.5,
1e-7);
   }
+
+  @Test
+  public void testBin() {
+    Assert.assertEquals(run("BIN(value, bounds)", ImmutableMap.of("value", 0, "bounds", ImmutableList.of(10,
20, 30))), 0);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 0)), 0);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 9)), 0);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 10)),
0);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 11)),
1);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 19)),
1);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 21)),
2);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 29)),
2);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 31)),
3);
+    Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 1000)),
3);
+  }
 }

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StatisticalBinningPerformanceDriver.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StatisticalBinningPerformanceDriver.java
b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StatisticalBinningPerformanceDriver.java
new file mode 100644
index 0000000..e6cc10b
--- /dev/null
+++ b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StatisticalBinningPerformanceDriver.java
@@ -0,0 +1,76 @@
+/*
+ *
+ *  Licensed to the Apache Software Foundation (ASF) under one
+ *  or more contributor license agreements.  See the NOTICE file
+ *  distributed with this work for additional information
+ *  regarding copyright ownership.  The ASF licenses this file
+ *  to you under the Apache License, Version 2.0 (the
+ *  "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ *
+ */
+package org.apache.metron.statistics;
+
+import com.google.common.collect.ImmutableList;
+import org.apache.commons.math3.random.GaussianRandomGenerator;
+import org.apache.commons.math3.random.MersenneTwister;
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+
+/**
+ * This is a driver to drive evaluation of the performance characteristics of the STATS_BIN
stellar function.
+ * It gets the distribution of the time it takes to calculate the bin of a million random
numbers against the quintile bins
+ * of a statistical distribution of 10000 normally distributed reals between [-1000, 1000].
+ *
+ * On my 4 year old macbook pro, the values came out to be
+ *
+ * Min/25th/50th/75th/Max Milliseconds: 2687.0 / 2700.5 / 2716.0 / 2733.5 / 3730.0
+ */
+public class StatisticalBinningPerformanceDriver {
+  public static int NUM_DATA_POINTS = 10000;
+  public static int NUM_RUNS = 30;
+  public static int TRIALS_PER_RUN = 1000000;
+  public static List<Number> PERCENTILES = ImmutableList.of(25.0, 50.0, 75.0);
+
+  public static void main(String... argv) {
+    DescriptiveStatistics perfStats = new DescriptiveStatistics();
+    OnlineStatisticsProvider statsProvider = new OnlineStatisticsProvider();
+    List<Double> values = new ArrayList<>();
+    GaussianRandomGenerator gaussian = new GaussianRandomGenerator(new MersenneTwister(0L));
+    for(int i = 0;i < NUM_DATA_POINTS;++i) {
+      //get the data point out of the [0,1] range
+      double d = 1000*gaussian.nextNormalizedDouble();
+      values.add(d);
+      statsProvider.addValue(d);
+    }
+
+    for(int perfRun = 0;perfRun < NUM_RUNS;++perfRun) {
+      StellarStatisticsFunctions.StatsBin bin = new StellarStatisticsFunctions.StatsBin();
+      long start = System.currentTimeMillis();
+      Random r = new Random(0);
+      for (int i = 0; i < TRIALS_PER_RUN; ++i) {
+        //grab a random value and fuzz it a bit so we make sure there's no cheating via caching
in t-digest.
+        bin.apply(ImmutableList.of(statsProvider, values.get(r.nextInt(values.size())) -
3.5, PERCENTILES));
+      }
+      perfStats.addValue(System.currentTimeMillis() - start);
+    }
+    System.out.println( "Min/25th/50th/75th/Max Milliseconds: "
+                      + perfStats.getMin()
+                      + " / " + perfStats.getPercentile(25)
+                      + " / " + perfStats.getPercentile(50)
+                      + " / " + perfStats.getPercentile(75)
+                      + " / " + perfStats.getMax()
+                      );
+  }
+}

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
index b9846e7..5eee84b 100644
--- a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
+++ b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
@@ -21,6 +21,7 @@
 package org.apache.metron.statistics;
 
 import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableList;
 import org.apache.commons.math3.random.GaussianRandomGenerator;
 import org.apache.commons.math3.random.MersenneTwister;
 import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
@@ -356,6 +357,56 @@ public class StellarStatisticsFunctionsTest {
     assertEquals(stats.getSkewness(), (Double) actual, 0.1);
   }
 
+  /**
+   * This test is testing that the STATS_BIN function operates correctly by taking a sorted
list of 
+   * numbers, walking down it and ensuring that the STATS_BIN for each number yields the
correct bin. 
+   * This is a reasonable test because we are not actually computing the bin so much as recognizing

+   * since the numbers are sorted, the bin will increase at the percentile boundaries, thus
we have 
+   * the expected bin without recreating the computation in the STATS_BIN function.
+   **/
+  @Test
+  public void testStatsBin() throws Exception {
+    statsInit(windowSize);
+    statsBinRunner(StellarStatisticsFunctions.StatsBin.BinSplits.QUARTILE.split);
+    statsBinRunner(StellarStatisticsFunctions.StatsBin.BinSplits.QUARTILE.split, "'QUARTILE'");
+    statsBinRunner(StellarStatisticsFunctions.StatsBin.BinSplits.QUINTILE.split, "'QUINTILE'");
+    statsBinRunner(StellarStatisticsFunctions.StatsBin.BinSplits.DECILE.split, "'DECILE'");
+    statsBinRunner(ImmutableList.of(25.0, 50.0, 75.0), "[25.0, 50.0, 75.0]");
+  }
+
+  @Test
+  public void testStatsBin_singleValue() throws Exception {
+    StatisticsProvider provider = (StatisticsProvider)run("STATS_INIT(" + windowSize + ")",
variables);
+    provider.addValue(10);
+    variables.put("stats", provider);
+    Assert.assertEquals(0, run(format("STATS_BIN(stats, %f)", 9.0), variables));
+    Assert.assertEquals(0, run(format("STATS_BIN(stats, %f)", 10.0), variables));
+    Assert.assertEquals(3, run(format("STATS_BIN(stats, %f)", 11.0), variables));
+  }
+
+  public void statsBinRunner(List<Number> splits) throws Exception {
+    statsBinRunner(splits, null);
+  }
+
+  public void statsBinRunner(List<Number> splits, String splitsName) throws Exception
{
+    int bin = 0;
+    StatisticsProvider provider = (StatisticsProvider)variables.get("stats");
+    for(Double d : stats.getSortedValues()) {
+      while ( bin < splits.size()  &&  d > provider.getPercentile(splits.get(bin).doubleValue())
) {
+        //increment the bin number until it includes the target value, or we run out of bins
+        bin++;
+      }
+
+      Object actual = null;
+      if(splitsName != null) {
+        actual = run(format("STATS_BIN(stats, %f, %s)", d, splitsName), variables);
+      }
+      else {
+        actual = run(format("STATS_BIN(stats, %f)", d), variables);
+      }
+      assertEquals(bin, actual);
+    }
+  }
 
   @Test
   public void testPercentileNoWindow() throws Exception {
@@ -384,4 +435,4 @@ public class StellarStatisticsFunctionsTest {
     actual = run("STATS_VARIANCE(null)", variables);
     assertTrue(((Double)actual).isNaN());
   }
-}
\ No newline at end of file
+}

http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-platform/metron-common/README.md
----------------------------------------------------------------------
diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md
index cf67b52..b41333b 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -43,85 +43,88 @@ Using parens such as: "foo" : "\<ok\>" requires escaping; "foo":
"\'\<ok\>\'"
 
 ## Stellar Core Functions
 
-|            |
-| ---------- |
-| [ `BLOOM_ADD`](#bloom_add)|
-| [ `BLOOM_EXISTS`](#bloom_exists)|
-| [ `BLOOM_INIT`](#bloom_init)|
-| [ `BLOOM_MERGE`](#bloom_merge)|
-| [ `DAY_OF_MONTH`](#day_of_month)|
-| [ `DAY_OF_WEEK`](#day_of_week)|
-| [ `DAY_OF_YEAR`](#day_of_year)|
-| [ `DOMAIN_REMOVE_SUBDOMAINS`](#domain_remove_subdomains)|
-| [ `DOMAIN_REMOVE_TLD`](#domain_remove_tld)|
-| [ `DOMAIN_TO_TLD`](#domain_to_tld)|
-| [ `ENDS_WITH`](#ends_with)|
-| [ `ENRICHMENT_EXISTS`](#enrichment_exists)|
-| [ `ENRICHMENT_GET`](#enrichment_get)|
-| [ `FILL_LEFT`](#fill_left)|
-| [ `FILL_RIGHT`](#fill_right)|
-| [ `GET`](#get)|
-| [ `GET_FIRST`](#get_first)|
-| [ `GET_LAST`](#get_last)|
-| [ `IN_SUBNET`](#in_subnet)|
-| [ `IS_DATE`](#is_date)|
-| [ `IS_DOMAIN`](#is_domain)|
-| [ `IS_EMAIL`](#is_email)|
-| [ `IS_EMPTY`](#is_empty)|
-| [ `IS_INTEGER`](#is_integer)|
-| [ `IS_IP`](#is_ip)|
-| [ `IS_URL`](#is_url)|
-| [ `JOIN`](#join)|
-| [ `KAFKA_GET`](#kafka_get)|
-| [ `KAFKA_PROPS`](#kafka_props)|
-| [ `KAFKA_PUT`](#kafka_put)|
-| [ `KAFKA_TAIL`](#kafka_tail)|
-| [ `LENGTH`](#length)|
-| [ `MAAS_GET_ENDPOINT`](#maas_get_endpoint)|
-| [ `MAAS_MODEL_APPLY`](#maas_model_apply)|
-| [ `MAP_EXISTS`](#map_exists)|
-| [ `MONTH`](#month)|
-| [ `PROFILE_GET`](#profile_get)|
-| [ `PROTOCOL_TO_NAME`](#protocol_to_name)|
-| [ `REGEXP_MATCH`](#regexp_match)|
-| [ `SPLIT`](#split)|
-| [ `STARTS_WITH`](#starts_with)|
-| [ `STATS_ADD`](../../metron-analytics/metron-statistics#stats_add)|
-| [ `STATS_COUNT`](../../metron-analytics/metron-statistics#stats_count)|
-| [ `STATS_GEOMETRIC_MEAN`](../../metron-analytics/metron-statistics#stats_geometric_mean)|
-| [ `STATS_INIT`](../../metron-analytics/metron-statistics#stats_init)|
-| [ `STATS_KURTOSIS`](../../metron-analytics/metron-statistics#stats_kurtosis)|
-| [ `STATS_MAX`](../../metron-analytics/metron-statistics#stats_max)|
-| [ `STATS_MEAN`](../../metron-analytics/metron-statistics#stats_mean)|
-| [ `STATS_MERGE`](../../metron-analytics/metron-statistics#stats_merge)|
-| [ `STATS_MIN`](../../metron-analytics/metron-statistics#stats_min)|
-| [ `STATS_PERCENTILE`](../../metron-analytics/metron-statistics#stats_percentile)|
-| [ `STATS_POPULATION_VARIANCE`](../../metron-analytics/metron-statistics#stats_population_variance)|
-| [ `STATS_QUADRATIC_MEAN`](../../metron-analytics/metron-statistics#stats_quadratic_mean)|
-| [ `STATS_SD`](../../metron-analytics/metron-statistics#stats_sd)|
-| [ `STATS_SKEWNESS`](../../metron-analytics/metron-statistics#stats_skewness)|
-| [ `STATS_SUM`](../../metron-analytics/metron-statistics#stats_sum)|
-| [ `STATS_SUM_LOGS`](../../metron-analytics/metron-statistics#stats_sum_logs)|
-| [ `STATS_SUM_SQUARES`](../../metron-analytics/metron-statistics#stats_sum_squares)|
-| [ `STATS_VARIANCE`](../../metron-analytics/metron-statistics#stats_variance)|
-| [ `SYSTEM_ENV_GET`](#system_env_get)|
-| [ `SYSTEM_PROPERTY_GET`](#system_property_get)|
-| [ `TO_DOUBLE`](#to_double)|
-| [ `TO_EPOCH_TIMESTAMP`](#to_epoch_timestamp)|
-| [ `TO_FLOAT`](#to_float)|
-| [ `TO_INTEGER`](#to_integer)|
-| [ `TO_LONG`](#to_long)|
-| [ `TO_LOWER`](#to_lower)|
-| [ `TO_STRING`](#to_string)|
-| [ `TO_UPPER`](#to_upper)|
-| [ `TRIM`](#trim)|
-| [ `URL_TO_HOST`](#url_to_host)|
-| [ `URL_TO_PATH`](#url_to_path)|
-| [ `URL_TO_PORT`](#url_to_port)|
-| [ `URL_TO_PROTOCOL`](#url_to_protocol)|
-| [ `WEEK_OF_MONTH`](#week_of_month)|
-| [ `WEEK_OF_YEAR`](#week_of_year)|
-| [ `YEAR`](#year)|
+|                                                                                       
            |
+| ----------                                                                            
            |
+| [ `ABS`](../../metron-analytics/metron-statistics#abs)                                
            |
+| [ `BIN`](../../metron-analytics/metron-statistics#bin)                                
            |
+| [ `BLOOM_ADD`](#bloom_add)                                                            
            |
+| [ `BLOOM_EXISTS`](#bloom_exists)                                                      
            |
+| [ `BLOOM_INIT`](#bloom_init)                                                          
            |
+| [ `BLOOM_MERGE`](#bloom_merge)                                                        
            |
+| [ `DAY_OF_MONTH`](#day_of_month)                                                      
            |
+| [ `DAY_OF_WEEK`](#day_of_week)                                                        
            |
+| [ `DAY_OF_YEAR`](#day_of_year)                                                        
            |
+| [ `DOMAIN_REMOVE_SUBDOMAINS`](#domain_remove_subdomains)                              
            |
+| [ `DOMAIN_REMOVE_TLD`](#domain_remove_tld)                                            
            |
+| [ `DOMAIN_TO_TLD`](#domain_to_tld)                                                    
            |
+| [ `ENDS_WITH`](#ends_with)                                                            
            |
+| [ `ENRICHMENT_EXISTS`](#enrichment_exists)                                            
            |
+| [ `ENRICHMENT_GET`](#enrichment_get)                                                  
            |
+| [ `FILL_LEFT`](#fill_left)                                                            
            |
+| [ `FILL_RIGHT`](#fill_right)                                                          
            |
+| [ `GET`](#get)                                                                        
            |
+| [ `GET_FIRST`](#get_first)                                                            
            |
+| [ `GET_LAST`](#get_last)                                                              
            |
+| [ `IN_SUBNET`](#in_subnet)                                                            
            |
+| [ `IS_DATE`](#is_date)                                                                
            |
+| [ `IS_DOMAIN`](#is_domain)                                                            
            |
+| [ `IS_EMAIL`](#is_email)                                                              
            |
+| [ `IS_EMPTY`](#is_empty)                                                              
            |
+| [ `IS_INTEGER`](#is_integer)                                                          
            |
+| [ `IS_IP`](#is_ip)                                                                    
            |
+| [ `IS_URL`](#is_url)                                                                  
            |
+| [ `JOIN`](#join)                                                                      
            |
+| [ `KAFKA_GET`](#kafka_get)                                                            
            |
+| [ `KAFKA_PROPS`](#kafka_props)                                                        
            |
+| [ `KAFKA_PUT`](#kafka_put)                                                            
            |
+| [ `KAFKA_TAIL`](#kafka_tail)                                                          
            |
+| [ `LENGTH`](#length)                                                                  
            |
+| [ `MAAS_GET_ENDPOINT`](#maas_get_endpoint)                                            
            |
+| [ `MAAS_MODEL_APPLY`](#maas_model_apply)                                              
            |
+| [ `MAP_EXISTS`](#map_exists)                                                          
            |
+| [ `MONTH`](#month)                                                                    
            |
+| [ `PROFILE_GET`](#profile_get)                                                        
            |
+| [ `PROTOCOL_TO_NAME`](#protocol_to_name)                                              
            |
+| [ `REGEXP_MATCH`](#regexp_match)                                                      
            |
+| [ `SPLIT`](#split)                                                                    
            |
+| [ `STARTS_WITH`](#starts_with)                                                        
            |
+| [ `STATS_ADD`](../../metron-analytics/metron-statistics#stats_add)                    
            |
+| [ `STATS_BIN`](../../metron-analytics/metron-statistics#stats_bin)                    
            |
+| [ `STATS_COUNT`](../../metron-analytics/metron-statistics#stats_count)                
            |
+| [ `STATS_GEOMETRIC_MEAN`](../../metron-analytics/metron-statistics#stats_geometric_mean)
          |
+| [ `STATS_INIT`](../../metron-analytics/metron-statistics#stats_init)                  
            |
+| [ `STATS_KURTOSIS`](../../metron-analytics/metron-statistics#stats_kurtosis)          
            |
+| [ `STATS_MAX`](../../metron-analytics/metron-statistics#stats_max)                    
            |
+| [ `STATS_MEAN`](../../metron-analytics/metron-statistics#stats_mean)                  
            |
+| [ `STATS_MERGE`](../../metron-analytics/metron-statistics#stats_merge)                
            |
+| [ `STATS_MIN`](../../metron-analytics/metron-statistics#stats_min)                    
            |
+| [ `STATS_PERCENTILE`](../../metron-analytics/metron-statistics#stats_percentile)      
            |
+| [ `STATS_POPULATION_VARIANCE`](../../metron-analytics/metron-statistics#stats_population_variance)
|
+| [ `STATS_QUADRATIC_MEAN`](../../metron-analytics/metron-statistics#stats_quadratic_mean)
          |
+| [ `STATS_SD`](../../metron-analytics/metron-statistics#stats_sd)                      
            |
+| [ `STATS_SKEWNESS`](../../metron-analytics/metron-statistics#stats_skewness)          
            |
+| [ `STATS_SUM`](../../metron-analytics/metron-statistics#stats_sum)                    
            |
+| [ `STATS_SUM_LOGS`](../../metron-analytics/metron-statistics#stats_sum_logs)          
            |
+| [ `STATS_SUM_SQUARES`](../../metron-analytics/metron-statistics#stats_sum_squares)    
            |
+| [ `STATS_VARIANCE`](../../metron-analytics/metron-statistics#stats_variance)          
            |
+| [ `SYSTEM_ENV_GET`](#system_env_get)                                                  
            |
+| [ `SYSTEM_PROPERTY_GET`](#system_property_get)                                        
            |
+| [ `TO_DOUBLE`](#to_double)                                                            
            |
+| [ `TO_EPOCH_TIMESTAMP`](#to_epoch_timestamp)                                          
            |
+| [ `TO_FLOAT`](#to_float)                                                              
            |
+| [ `TO_INTEGER`](#to_integer)                                                          
            |
+| [ `TO_LONG`](#to_long)                                                                
            |
+| [ `TO_LOWER`](#to_lower)                                                              
            |
+| [ `TO_STRING`](#to_string)                                                            
            |
+| [ `TO_UPPER`](#to_upper)                                                              
            |
+| [ `TRIM`](#trim)                                                                      
            |
+| [ `URL_TO_HOST`](#url_to_host)                                                        
            |
+| [ `URL_TO_PATH`](#url_to_path)                                                        
            |
+| [ `URL_TO_PORT`](#url_to_port)                                                        
            |
+| [ `URL_TO_PROTOCOL`](#url_to_protocol)                                                
            |
+| [ `WEEK_OF_MONTH`](#week_of_month)                                                    
            |
+| [ `WEEK_OF_YEAR`](#week_of_year)                                                      
            |
+| [ `YEAR`](#year)                                                                      
            |
 
 ### `BLOOM_ADD`
   * Description: Adds an element to the bloom filter passed in


Mime
View raw message