Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 731D7200BFB for ; Wed, 11 Jan 2017 15:11:44 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 721F1160B4E; Wed, 11 Jan 2017 14:11:44 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F3751160B2E for ; Wed, 11 Jan 2017 15:11:42 +0100 (CET) Received: (qmail 23361 invoked by uid 500); 11 Jan 2017 14:11:42 -0000 Mailing-List: contact commits-help@metron.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@metron.incubator.apache.org Delivered-To: mailing list commits@metron.incubator.apache.org Received: (qmail 23352 invoked by uid 99); 11 Jan 2017 14:11:42 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Jan 2017 14:11:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id AD2921A0093 for ; Wed, 11 Jan 2017 14:11:41 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id vejKzPeqDeyt for ; Wed, 11 Jan 2017 14:11:36 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id 9E33C5FE11 for ; Wed, 11 Jan 2017 14:11:35 +0000 (UTC) Received: (qmail 23225 invoked by uid 99); 11 Jan 2017 14:11:35 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 Jan 2017 14:11:35 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id D23B5DFA22; Wed, 11 Jan 2017 14:11:34 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: cestella@apache.org To: commits@metron.incubator.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: incubator-metron git commit: METRON-637: Add a STATS_BIN function to Stellar. closes apache/incubator-metron#401 Date: Wed, 11 Jan 2017 14:11:34 +0000 (UTC) archived-at: Wed, 11 Jan 2017 14:11:44 -0000 Repository: incubator-metron Updated Branches: refs/heads/master e7fcc755c -> 09cb50288 METRON-637: Add a STATS_BIN function to Stellar. closes apache/incubator-metron#401 Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/09cb5028 Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/09cb5028 Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/09cb5028 Branch: refs/heads/master Commit: 09cb50288f3044e30e6674fca03ef2d05483cff8 Parents: e7fcc75 Author: cstella Authored: Wed Jan 11 09:11:25 2017 -0500 Committer: cstella Committed: Wed Jan 11 09:11:25 2017 -0500 ---------------------------------------------------------------------- metron-analytics/metron-statistics/README.md | 21 ++- .../apache/metron/statistics/MathFunctions.java | 50 +++++- .../statistics/StellarStatisticsFunctions.java | 56 +++++++ .../metron/statistics/MathFunctionsTest.java | 15 ++ .../StatisticalBinningPerformanceDriver.java | 76 +++++++++ .../StellarStatisticsFunctionsTest.java | 53 +++++- metron-platform/metron-common/README.md | 161 ++++++++++--------- 7 files changed, 346 insertions(+), 86 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/README.md ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-statistics/README.md b/metron-analytics/metron-statistics/README.md index edc68e1..346179b 100644 --- a/metron-analytics/metron-statistics/README.md +++ b/metron-analytics/metron-statistics/README.md @@ -11,11 +11,19 @@ functions can be used from everywhere where Stellar is used. ## Stellar Functions ### Mathematical Functions + #### `ABS` * Description: Returns the absolute value of a number. * Input: * number - The number to take the absolute value of * Returns: The absolute value of the number passed in. +* +#### `BIN` +* Description: Computes the bin that the value is in given a set of bounds. +* Input: + * value - The value to bin + * bounds - A list of value bounds (excluding min and max) in sorted order. +* Returns: Which bin N the value falls in such that bound(N-1) < value <= bound(N). No min and max bounds are provided, so values smaller than the 0'th bound go in the 0'th bin, and values greater than the last bound go in the M'th bin. ### Distributional Statistics @@ -27,6 +35,14 @@ functions can be used from everywhere where Stellar is used. * value+ - One or more numbers to add * Returns: A Stellar statistics object +#### `STATS_BIN` + * Description: Computes the bin that the value is in based on the statistical distribution. + * Input: + * stats - The Stellar statistics object + * value - The value to bin + * bounds? - A list of percentile bin bounds (excluding min and max) or a string representing a known and common set of bins. For convenience, we have provided QUARTILE, QUINTILE, and DECILE which you can pass in as a string arg. If this argument is omitted, then we assume a Quartile bin split. + * Returns: "Which bin N the value falls in such that bound(N-1) < value <= bound(N). No min and max bounds are provided, so values smaller than the 0'th bound go in the 0'th bin, and values greater than the last bound go in the M'th bin. + #### `STATS_COUNT` * Description: Calculates the count of the values accumulated (or in the window if a window is used). * Input: @@ -129,8 +145,9 @@ functions can be used from everywhere where Stellar is used. * Input: * stats - The Stellar statistics object * Returns: The variance of the values in the window or NaN if the statistics object is null. - - ### Statistical Outlier Detection + + +### Statistical Outlier Detection #### `OUTLIER_MAD_STATE_MERGE` * Description: Update the statistical state required to compute the Median Absolute Deviation. http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java index 548a611..9364095 100644 --- a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java +++ b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/MathFunctions.java @@ -19,12 +19,13 @@ */ package org.apache.metron.statistics; -import org.apache.metron.common.dsl.Context; -import org.apache.metron.common.dsl.ParseException; -import org.apache.metron.common.dsl.Stellar; -import org.apache.metron.common.dsl.StellarFunction; +import org.apache.metron.common.dsl.*; +import java.util.ArrayList; import java.util.List; +import java.util.function.Function; + +import static org.apache.metron.common.utils.ConversionUtils.convert; public class MathFunctions { @@ -59,4 +60,45 @@ public class MathFunctions { return true; } } + + /** + * Calculates the statistical bin that a value falls in. + */ + @Stellar(name = "BIN" + , description = "Computes the bin that the value is in given a set of bounds." + , params = { + "value - The value to bin" + , "bounds - A list of value bounds (excluding min and max) in sorted order." + } + ,returns = "Which bin N the value falls in such that bound(N-1) < value <= bound(N). " + + "No min and max bounds are provided, so values smaller than the 0'th bound go in the 0'th bin, " + + "and values greater than the last bound go in the M'th bin." + ) + public static class Bin extends BaseStellarFunction { + + public static int getBin(double value, int numBins, Function boundFunc) { + double lastBound = Double.NEGATIVE_INFINITY; + for(int bin = 0; bin < numBins;++bin) { + double bound = boundFunc.apply(bin); + if(lastBound > bound ) { + throw new IllegalStateException("Your bins must be non-decreasing"); + } + if(value <= bound) { + return bin; + } + lastBound = bound; + } + return numBins; + } + + @Override + public Object apply(List args) { + Double value = convert(args.get(0), Double.class); + final List bins = args.size() > 1?convert(args.get(1), List.class):null; + if ( value == null || bins == null || bins.size() == 0) { + return -1; + } + return getBin(value, bins.size(), bin -> bins.get(bin).doubleValue()); + } + } } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java index 2ea8ee1..2223b47 100644 --- a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java +++ b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java @@ -20,9 +20,12 @@ package org.apache.metron.statistics; +import com.google.common.collect.ImmutableList; import org.apache.metron.common.dsl.BaseStellarFunction; import org.apache.metron.common.dsl.Stellar; +import org.apache.metron.common.utils.ConversionUtils; +import java.util.ArrayList; import java.util.Collections; import java.util.List; @@ -425,4 +428,57 @@ public class StellarStatisticsFunctions { return result; } } + + /** + * Calculates the statistical bin that a value falls in. + */ + @Stellar(namespace = "STATS", name = "BIN" + , description = "Computes the bin that the value is in based on the statistical distribution." + , params = { + "stats - The Stellar statistics object" + , "value - The value to bin" + , "bounds? - A list of percentile bin bounds (excluding min and max) or a string representing a known and common set of bins. " + + "For convenience, we have provided QUARTILE, QUINTILE, and DECILE which you can pass in as a string arg." + + " If this argument is omitted, then we assume a Quartile bin split." + } + ,returns = "Which bin N the value falls in such that bound(N-1) < value <= bound(N). " + + "No min and max bounds are provided, so values smaller than the 0'th bound go in the 0'th bin, " + + "and values greater than the last bound go in the M'th bin." + ) + public static class StatsBin extends BaseStellarFunction { + public enum BinSplits { + QUARTILE(ImmutableList.of(25.0, 50.0, 75.0)), + QUINTILE(ImmutableList.of(20.0, 40.0, 60.0, 80.0)), + DECILE(ImmutableList.of(10.0, 20.0, 30.0, 40.0, 50.0, 60.0, 70.0, 80.0, 90.0)) + ; + public final List split; + BinSplits(List split) { + this.split = split; + } + + public static List getSplit(Object o) { + if(o instanceof String) { + return BinSplits.valueOf((String)o).split; + } + else if(o instanceof List) { + return ConversionUtils.convert(o, List.class); + } + throw new IllegalStateException("The split you tried to pass is not a valid split: " + o.toString()); + } + } + + + @Override + public Object apply(List args) { + StatisticsProvider stats = convert(args.get(0), StatisticsProvider.class); + Double value = convert(args.get(1), Double.class); + final List bins = args.size() > 2?BinSplits.getSplit(args.get(2)):BinSplits.QUARTILE.split; + + if (stats == null || value == null || bins.size() == 0) { + return -1; + } + return MathFunctions.Bin.getBin(value, bins.size(), bin -> stats.getPercentile(bins.get(bin).doubleValue())); + } + } + } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java index 91e4de5..c615966 100644 --- a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java +++ b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/MathFunctionsTest.java @@ -19,6 +19,7 @@ */ package org.apache.metron.statistics; +import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import org.apache.metron.common.dsl.Context; import org.apache.metron.common.dsl.StellarFunctions; @@ -43,4 +44,18 @@ public class MathFunctionsTest { Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", 10.5)), 10.5, 1e-7); Assert.assertEquals((Double)run("ABS(value)", ImmutableMap.of("value", -10.5)), 10.5, 1e-7); } + + @Test + public void testBin() { + Assert.assertEquals(run("BIN(value, bounds)", ImmutableMap.of("value", 0, "bounds", ImmutableList.of(10, 20, 30))), 0); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 0)), 0); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 9)), 0); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 10)), 0); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 11)), 1); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 19)), 1); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 21)), 2); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 29)), 2); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 31)), 3); + Assert.assertEquals(run("BIN(value, [ 10, 20, 30 ])", ImmutableMap.of("value", 1000)), 3); + } } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StatisticalBinningPerformanceDriver.java ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StatisticalBinningPerformanceDriver.java b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StatisticalBinningPerformanceDriver.java new file mode 100644 index 0000000..e6cc10b --- /dev/null +++ b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StatisticalBinningPerformanceDriver.java @@ -0,0 +1,76 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ +package org.apache.metron.statistics; + +import com.google.common.collect.ImmutableList; +import org.apache.commons.math3.random.GaussianRandomGenerator; +import org.apache.commons.math3.random.MersenneTwister; +import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; + +import java.util.ArrayList; +import java.util.List; +import java.util.Random; + +/** + * This is a driver to drive evaluation of the performance characteristics of the STATS_BIN stellar function. + * It gets the distribution of the time it takes to calculate the bin of a million random numbers against the quintile bins + * of a statistical distribution of 10000 normally distributed reals between [-1000, 1000]. + * + * On my 4 year old macbook pro, the values came out to be + * + * Min/25th/50th/75th/Max Milliseconds: 2687.0 / 2700.5 / 2716.0 / 2733.5 / 3730.0 + */ +public class StatisticalBinningPerformanceDriver { + public static int NUM_DATA_POINTS = 10000; + public static int NUM_RUNS = 30; + public static int TRIALS_PER_RUN = 1000000; + public static List PERCENTILES = ImmutableList.of(25.0, 50.0, 75.0); + + public static void main(String... argv) { + DescriptiveStatistics perfStats = new DescriptiveStatistics(); + OnlineStatisticsProvider statsProvider = new OnlineStatisticsProvider(); + List values = new ArrayList<>(); + GaussianRandomGenerator gaussian = new GaussianRandomGenerator(new MersenneTwister(0L)); + for(int i = 0;i < NUM_DATA_POINTS;++i) { + //get the data point out of the [0,1] range + double d = 1000*gaussian.nextNormalizedDouble(); + values.add(d); + statsProvider.addValue(d); + } + + for(int perfRun = 0;perfRun < NUM_RUNS;++perfRun) { + StellarStatisticsFunctions.StatsBin bin = new StellarStatisticsFunctions.StatsBin(); + long start = System.currentTimeMillis(); + Random r = new Random(0); + for (int i = 0; i < TRIALS_PER_RUN; ++i) { + //grab a random value and fuzz it a bit so we make sure there's no cheating via caching in t-digest. + bin.apply(ImmutableList.of(statsProvider, values.get(r.nextInt(values.size())) - 3.5, PERCENTILES)); + } + perfStats.addValue(System.currentTimeMillis() - start); + } + System.out.println( "Min/25th/50th/75th/Max Milliseconds: " + + perfStats.getMin() + + " / " + perfStats.getPercentile(25) + + " / " + perfStats.getPercentile(50) + + " / " + perfStats.getPercentile(75) + + " / " + perfStats.getMax() + ); + } +} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java ---------------------------------------------------------------------- diff --git a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java index b9846e7..5eee84b 100644 --- a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java +++ b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java @@ -21,6 +21,7 @@ package org.apache.metron.statistics; import com.google.common.base.Joiner; +import com.google.common.collect.ImmutableList; import org.apache.commons.math3.random.GaussianRandomGenerator; import org.apache.commons.math3.random.MersenneTwister; import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; @@ -356,6 +357,56 @@ public class StellarStatisticsFunctionsTest { assertEquals(stats.getSkewness(), (Double) actual, 0.1); } + /** + * This test is testing that the STATS_BIN function operates correctly by taking a sorted list of + * numbers, walking down it and ensuring that the STATS_BIN for each number yields the correct bin. + * This is a reasonable test because we are not actually computing the bin so much as recognizing + * since the numbers are sorted, the bin will increase at the percentile boundaries, thus we have + * the expected bin without recreating the computation in the STATS_BIN function. + **/ + @Test + public void testStatsBin() throws Exception { + statsInit(windowSize); + statsBinRunner(StellarStatisticsFunctions.StatsBin.BinSplits.QUARTILE.split); + statsBinRunner(StellarStatisticsFunctions.StatsBin.BinSplits.QUARTILE.split, "'QUARTILE'"); + statsBinRunner(StellarStatisticsFunctions.StatsBin.BinSplits.QUINTILE.split, "'QUINTILE'"); + statsBinRunner(StellarStatisticsFunctions.StatsBin.BinSplits.DECILE.split, "'DECILE'"); + statsBinRunner(ImmutableList.of(25.0, 50.0, 75.0), "[25.0, 50.0, 75.0]"); + } + + @Test + public void testStatsBin_singleValue() throws Exception { + StatisticsProvider provider = (StatisticsProvider)run("STATS_INIT(" + windowSize + ")", variables); + provider.addValue(10); + variables.put("stats", provider); + Assert.assertEquals(0, run(format("STATS_BIN(stats, %f)", 9.0), variables)); + Assert.assertEquals(0, run(format("STATS_BIN(stats, %f)", 10.0), variables)); + Assert.assertEquals(3, run(format("STATS_BIN(stats, %f)", 11.0), variables)); + } + + public void statsBinRunner(List splits) throws Exception { + statsBinRunner(splits, null); + } + + public void statsBinRunner(List splits, String splitsName) throws Exception { + int bin = 0; + StatisticsProvider provider = (StatisticsProvider)variables.get("stats"); + for(Double d : stats.getSortedValues()) { + while ( bin < splits.size() && d > provider.getPercentile(splits.get(bin).doubleValue()) ) { + //increment the bin number until it includes the target value, or we run out of bins + bin++; + } + + Object actual = null; + if(splitsName != null) { + actual = run(format("STATS_BIN(stats, %f, %s)", d, splitsName), variables); + } + else { + actual = run(format("STATS_BIN(stats, %f)", d), variables); + } + assertEquals(bin, actual); + } + } @Test public void testPercentileNoWindow() throws Exception { @@ -384,4 +435,4 @@ public class StellarStatisticsFunctionsTest { actual = run("STATS_VARIANCE(null)", variables); assertTrue(((Double)actual).isNaN()); } -} \ No newline at end of file +} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/09cb5028/metron-platform/metron-common/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md index cf67b52..b41333b 100644 --- a/metron-platform/metron-common/README.md +++ b/metron-platform/metron-common/README.md @@ -43,85 +43,88 @@ Using parens such as: "foo" : "\" requires escaping; "foo": "\'\\'" ## Stellar Core Functions -| | -| ---------- | -| [ `BLOOM_ADD`](#bloom_add)| -| [ `BLOOM_EXISTS`](#bloom_exists)| -| [ `BLOOM_INIT`](#bloom_init)| -| [ `BLOOM_MERGE`](#bloom_merge)| -| [ `DAY_OF_MONTH`](#day_of_month)| -| [ `DAY_OF_WEEK`](#day_of_week)| -| [ `DAY_OF_YEAR`](#day_of_year)| -| [ `DOMAIN_REMOVE_SUBDOMAINS`](#domain_remove_subdomains)| -| [ `DOMAIN_REMOVE_TLD`](#domain_remove_tld)| -| [ `DOMAIN_TO_TLD`](#domain_to_tld)| -| [ `ENDS_WITH`](#ends_with)| -| [ `ENRICHMENT_EXISTS`](#enrichment_exists)| -| [ `ENRICHMENT_GET`](#enrichment_get)| -| [ `FILL_LEFT`](#fill_left)| -| [ `FILL_RIGHT`](#fill_right)| -| [ `GET`](#get)| -| [ `GET_FIRST`](#get_first)| -| [ `GET_LAST`](#get_last)| -| [ `IN_SUBNET`](#in_subnet)| -| [ `IS_DATE`](#is_date)| -| [ `IS_DOMAIN`](#is_domain)| -| [ `IS_EMAIL`](#is_email)| -| [ `IS_EMPTY`](#is_empty)| -| [ `IS_INTEGER`](#is_integer)| -| [ `IS_IP`](#is_ip)| -| [ `IS_URL`](#is_url)| -| [ `JOIN`](#join)| -| [ `KAFKA_GET`](#kafka_get)| -| [ `KAFKA_PROPS`](#kafka_props)| -| [ `KAFKA_PUT`](#kafka_put)| -| [ `KAFKA_TAIL`](#kafka_tail)| -| [ `LENGTH`](#length)| -| [ `MAAS_GET_ENDPOINT`](#maas_get_endpoint)| -| [ `MAAS_MODEL_APPLY`](#maas_model_apply)| -| [ `MAP_EXISTS`](#map_exists)| -| [ `MONTH`](#month)| -| [ `PROFILE_GET`](#profile_get)| -| [ `PROTOCOL_TO_NAME`](#protocol_to_name)| -| [ `REGEXP_MATCH`](#regexp_match)| -| [ `SPLIT`](#split)| -| [ `STARTS_WITH`](#starts_with)| -| [ `STATS_ADD`](../../metron-analytics/metron-statistics#stats_add)| -| [ `STATS_COUNT`](../../metron-analytics/metron-statistics#stats_count)| -| [ `STATS_GEOMETRIC_MEAN`](../../metron-analytics/metron-statistics#stats_geometric_mean)| -| [ `STATS_INIT`](../../metron-analytics/metron-statistics#stats_init)| -| [ `STATS_KURTOSIS`](../../metron-analytics/metron-statistics#stats_kurtosis)| -| [ `STATS_MAX`](../../metron-analytics/metron-statistics#stats_max)| -| [ `STATS_MEAN`](../../metron-analytics/metron-statistics#stats_mean)| -| [ `STATS_MERGE`](../../metron-analytics/metron-statistics#stats_merge)| -| [ `STATS_MIN`](../../metron-analytics/metron-statistics#stats_min)| -| [ `STATS_PERCENTILE`](../../metron-analytics/metron-statistics#stats_percentile)| -| [ `STATS_POPULATION_VARIANCE`](../../metron-analytics/metron-statistics#stats_population_variance)| -| [ `STATS_QUADRATIC_MEAN`](../../metron-analytics/metron-statistics#stats_quadratic_mean)| -| [ `STATS_SD`](../../metron-analytics/metron-statistics#stats_sd)| -| [ `STATS_SKEWNESS`](../../metron-analytics/metron-statistics#stats_skewness)| -| [ `STATS_SUM`](../../metron-analytics/metron-statistics#stats_sum)| -| [ `STATS_SUM_LOGS`](../../metron-analytics/metron-statistics#stats_sum_logs)| -| [ `STATS_SUM_SQUARES`](../../metron-analytics/metron-statistics#stats_sum_squares)| -| [ `STATS_VARIANCE`](../../metron-analytics/metron-statistics#stats_variance)| -| [ `SYSTEM_ENV_GET`](#system_env_get)| -| [ `SYSTEM_PROPERTY_GET`](#system_property_get)| -| [ `TO_DOUBLE`](#to_double)| -| [ `TO_EPOCH_TIMESTAMP`](#to_epoch_timestamp)| -| [ `TO_FLOAT`](#to_float)| -| [ `TO_INTEGER`](#to_integer)| -| [ `TO_LONG`](#to_long)| -| [ `TO_LOWER`](#to_lower)| -| [ `TO_STRING`](#to_string)| -| [ `TO_UPPER`](#to_upper)| -| [ `TRIM`](#trim)| -| [ `URL_TO_HOST`](#url_to_host)| -| [ `URL_TO_PATH`](#url_to_path)| -| [ `URL_TO_PORT`](#url_to_port)| -| [ `URL_TO_PROTOCOL`](#url_to_protocol)| -| [ `WEEK_OF_MONTH`](#week_of_month)| -| [ `WEEK_OF_YEAR`](#week_of_year)| -| [ `YEAR`](#year)| +| | +| ---------- | +| [ `ABS`](../../metron-analytics/metron-statistics#abs) | +| [ `BIN`](../../metron-analytics/metron-statistics#bin) | +| [ `BLOOM_ADD`](#bloom_add) | +| [ `BLOOM_EXISTS`](#bloom_exists) | +| [ `BLOOM_INIT`](#bloom_init) | +| [ `BLOOM_MERGE`](#bloom_merge) | +| [ `DAY_OF_MONTH`](#day_of_month) | +| [ `DAY_OF_WEEK`](#day_of_week) | +| [ `DAY_OF_YEAR`](#day_of_year) | +| [ `DOMAIN_REMOVE_SUBDOMAINS`](#domain_remove_subdomains) | +| [ `DOMAIN_REMOVE_TLD`](#domain_remove_tld) | +| [ `DOMAIN_TO_TLD`](#domain_to_tld) | +| [ `ENDS_WITH`](#ends_with) | +| [ `ENRICHMENT_EXISTS`](#enrichment_exists) | +| [ `ENRICHMENT_GET`](#enrichment_get) | +| [ `FILL_LEFT`](#fill_left) | +| [ `FILL_RIGHT`](#fill_right) | +| [ `GET`](#get) | +| [ `GET_FIRST`](#get_first) | +| [ `GET_LAST`](#get_last) | +| [ `IN_SUBNET`](#in_subnet) | +| [ `IS_DATE`](#is_date) | +| [ `IS_DOMAIN`](#is_domain) | +| [ `IS_EMAIL`](#is_email) | +| [ `IS_EMPTY`](#is_empty) | +| [ `IS_INTEGER`](#is_integer) | +| [ `IS_IP`](#is_ip) | +| [ `IS_URL`](#is_url) | +| [ `JOIN`](#join) | +| [ `KAFKA_GET`](#kafka_get) | +| [ `KAFKA_PROPS`](#kafka_props) | +| [ `KAFKA_PUT`](#kafka_put) | +| [ `KAFKA_TAIL`](#kafka_tail) | +| [ `LENGTH`](#length) | +| [ `MAAS_GET_ENDPOINT`](#maas_get_endpoint) | +| [ `MAAS_MODEL_APPLY`](#maas_model_apply) | +| [ `MAP_EXISTS`](#map_exists) | +| [ `MONTH`](#month) | +| [ `PROFILE_GET`](#profile_get) | +| [ `PROTOCOL_TO_NAME`](#protocol_to_name) | +| [ `REGEXP_MATCH`](#regexp_match) | +| [ `SPLIT`](#split) | +| [ `STARTS_WITH`](#starts_with) | +| [ `STATS_ADD`](../../metron-analytics/metron-statistics#stats_add) | +| [ `STATS_BIN`](../../metron-analytics/metron-statistics#stats_bin) | +| [ `STATS_COUNT`](../../metron-analytics/metron-statistics#stats_count) | +| [ `STATS_GEOMETRIC_MEAN`](../../metron-analytics/metron-statistics#stats_geometric_mean) | +| [ `STATS_INIT`](../../metron-analytics/metron-statistics#stats_init) | +| [ `STATS_KURTOSIS`](../../metron-analytics/metron-statistics#stats_kurtosis) | +| [ `STATS_MAX`](../../metron-analytics/metron-statistics#stats_max) | +| [ `STATS_MEAN`](../../metron-analytics/metron-statistics#stats_mean) | +| [ `STATS_MERGE`](../../metron-analytics/metron-statistics#stats_merge) | +| [ `STATS_MIN`](../../metron-analytics/metron-statistics#stats_min) | +| [ `STATS_PERCENTILE`](../../metron-analytics/metron-statistics#stats_percentile) | +| [ `STATS_POPULATION_VARIANCE`](../../metron-analytics/metron-statistics#stats_population_variance) | +| [ `STATS_QUADRATIC_MEAN`](../../metron-analytics/metron-statistics#stats_quadratic_mean) | +| [ `STATS_SD`](../../metron-analytics/metron-statistics#stats_sd) | +| [ `STATS_SKEWNESS`](../../metron-analytics/metron-statistics#stats_skewness) | +| [ `STATS_SUM`](../../metron-analytics/metron-statistics#stats_sum) | +| [ `STATS_SUM_LOGS`](../../metron-analytics/metron-statistics#stats_sum_logs) | +| [ `STATS_SUM_SQUARES`](../../metron-analytics/metron-statistics#stats_sum_squares) | +| [ `STATS_VARIANCE`](../../metron-analytics/metron-statistics#stats_variance) | +| [ `SYSTEM_ENV_GET`](#system_env_get) | +| [ `SYSTEM_PROPERTY_GET`](#system_property_get) | +| [ `TO_DOUBLE`](#to_double) | +| [ `TO_EPOCH_TIMESTAMP`](#to_epoch_timestamp) | +| [ `TO_FLOAT`](#to_float) | +| [ `TO_INTEGER`](#to_integer) | +| [ `TO_LONG`](#to_long) | +| [ `TO_LOWER`](#to_lower) | +| [ `TO_STRING`](#to_string) | +| [ `TO_UPPER`](#to_upper) | +| [ `TRIM`](#trim) | +| [ `URL_TO_HOST`](#url_to_host) | +| [ `URL_TO_PATH`](#url_to_path) | +| [ `URL_TO_PORT`](#url_to_port) | +| [ `URL_TO_PROTOCOL`](#url_to_protocol) | +| [ `WEEK_OF_MONTH`](#week_of_month) | +| [ `WEEK_OF_YEAR`](#week_of_year) | +| [ `YEAR`](#year) | ### `BLOOM_ADD` * Description: Adds an element to the bloom filter passed in