metron-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickal...@apache.org
Subject metron git commit: METRON-1132 Enhance Profiler Debug Functions (nickwallen) closes apache/metron#716
Date Thu, 31 Aug 2017 14:38:50 GMT
Repository: metron
Updated Branches:
  refs/heads/master 1a4f19c86 -> c73c7af1d


METRON-1132 Enhance Profiler Debug Functions (nickwallen) closes apache/metron#716


Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/c73c7af1
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/c73c7af1
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/c73c7af1

Branch: refs/heads/master
Commit: c73c7af1dc3cb6502517bc7ac1d37c631ec3dbe5
Parents: 1a4f19c
Author: nickwallen <nick@nickallen.org>
Authored: Thu Aug 31 10:38:21 2017 -0400
Committer: nickallen <nickallen@apache.org>
Committed: Thu Aug 31 10:38:21 2017 -0400

----------------------------------------------------------------------
 .../metron-profiler-client/README.md            | 21 +++--
 .../client/stellar/ProfilerFunctions.java       | 35 ++++++--
 .../client/stellar/ProfilerFunctionsTest.java   | 93 ++++++++++++++++++--
 .../metron/profiler/StandAloneProfiler.java     | 40 +++++++++
 metron-analytics/metron-profiler/README.md      |  2 +
 5 files changed, 172 insertions(+), 19 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler-client/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/README.md b/metron-analytics/metron-profiler-client/README.md
index a8b5a55..27aa3f4 100644
--- a/metron-analytics/metron-profiler-client/README.md
+++ b/metron-analytics/metron-profiler-client/README.md
@@ -418,8 +418,11 @@ Follow these steps in the Stellar REPL to see how it can be used to help
create
     ```
     [Stellar]>>> profiler := PROFILER_INIT(conf)
     [Stellar]>>> profiler
-    org.apache.metron.profiler.StandAloneProfiler@4f8ef473
+    Profiler{1 profile(s), 0 messages(s), 0 route(s)}
     ```
+    The profiler itself will show the number of profiles defined, the number of messages
applied, and the number of routes taken.  
+    
+    A route is defined when a message is applied to a specific profile.  If a message is
applied and not needed by any profile, then there are no routes.  If a message is needed by
one profile, then one route has been defined.  If a message is needed by two profiles, then
two routes have been defined.  
 
 1. Create a message to simulate the type of telemetry that you expect to be profiled.   As
an example, in the editor copy/paste the JSON below.
     ```
@@ -436,14 +439,18 @@ Follow these steps in the Stellar REPL to see how it can be used to
help create
 1. Apply some telemetry messages to your profiles.  The following applies the same message
3 times.
     ```
     [Stellar]>>> PROFILER_APPLY(message, profiler)
-    org.apache.metron.profiler.StandAloneProfiler@4f8ef473
-
+    Profiler{1 profile(s), 1 messages(s), 1 route(s)}
+    ```
+    ```
     [Stellar]>>> PROFILER_APPLY(message, profiler)
-    org.apache.metron.profiler.StandAloneProfiler@4f8ef473
-
+    Profiler{1 profile(s), 2 messages(s), 2 route(s)}
+    ```
+    ```
     [Stellar]>>> PROFILER_APPLY(message, profiler)
-    org.apache.metron.profiler.StandAloneProfiler@4f8ef473
+    Profiler{1 profile(s), 3 messages(s), 3 route(s)}
     ```
+    
+    It is also possible to apply multiple messages at once.  This is useful when testing
against a larger set of data.  To do this, create a string that contains a JSON array of messages
and pass that to the `PROFILER_APPLY` function.
 
 1. Flush the Profiler to see what has been calculated.  A flush is what occurs at the end
of each 15 minute period in the Profiler.  The result is a list of profile measurements. 
Each measurement is a map containing detailed information about the profile data that has
been generated.
     ```
@@ -455,4 +462,4 @@ Follow these steps in the Stellar REPL to see how it can be used to help
create
     
     This profile simply counts the number of messages by IP source address.  Notice that
the value is '3' for the entity '10.0.0.1' as we applied 3 messages with an 'ip_src_addr'
of '10.0.0.1'.  There will always be one measurement for each [profile, entity] pair.
     
-1. If you are unhappy with the data that has been generated, then 'wash, rinse and repeat'
this process.  After you are satisfied with the data being generated by the profile, then
follow the [Getting Started](../metron-profiler#getting-started) guide to use the profile
against your live, streaming data in a Metron cluster.
\ No newline at end of file
+1. If you are unhappy with the data that has been generated, then 'wash, rinse and repeat'
this process.  Once you are happy with the profile that was created, follow the [Getting Started](../metron-profiler#getting-started)
guide to use the profile against your live, streaming data in a Metron cluster.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java
b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java
index 827e1c4..8df5ca8 100644
--- a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java
+++ b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerFunctions.java
@@ -28,6 +28,8 @@ import org.apache.metron.stellar.dsl.Context;
 import org.apache.metron.stellar.dsl.ParseException;
 import org.apache.metron.stellar.dsl.Stellar;
 import org.apache.metron.stellar.dsl.StellarFunction;
+import org.apache.storm.shade.org.apache.commons.lang.ClassUtils;
+import org.json.simple.JSONArray;
 import org.json.simple.JSONObject;
 import org.json.simple.parser.JSONParser;
 import org.slf4j.LoggerFactory;
@@ -107,7 +109,7 @@ public class ProfilerFunctions {
           name="APPLY",
           description="Apply a message to a local profile runner.",
           params={
-                  "message", "The message to apply.",
+                  "message(s)", "The message to apply.  A JSON list can be used to apply
multiple messages.",
                   "profiler", "A local profile runner returned by PROFILER_INIT."
           },
           returns="The local profile runner."
@@ -129,16 +131,33 @@ public class ProfilerFunctions {
     @Override
     public Object apply(List<Object> args, Context context) throws ParseException {
 
-      // user must provide the json telemetry message
+      // user must provide the message as a string
       String arg0 = Util.getArg(0, String.class, args);
       if(arg0 == null) {
         throw new IllegalArgumentException(format("expected string, found null"));
       }
 
-      // parse the message
-      JSONObject message;
+      // there could be one or more messages
+      List<JSONObject> messages = new ArrayList<>();
       try {
-        message = (JSONObject) parser.parse(arg0);
+        Object parsedArg0 = parser.parse(arg0);
+        if(parsedArg0 instanceof JSONObject) {
+          // if there is only one message
+          messages.add((JSONObject) parsedArg0);
+
+        } else if(parsedArg0 instanceof JSONArray) {
+          // there are multiple messages
+          JSONArray jsonArray = (JSONArray) parsedArg0;
+          for(Object json: jsonArray) {
+            if(json instanceof JSONObject) {
+              messages.add((JSONObject) json);
+
+            } else {
+              throw new IllegalArgumentException(format("invalid message: found '%s', expected
JSONObject",
+                              ClassUtils.getShortClassName(json, "null")));
+            }
+          }
+        }
 
       } catch(org.json.simple.parser.ParseException e) {
         throw new IllegalArgumentException("invalid message", e);
@@ -147,10 +166,12 @@ public class ProfilerFunctions {
       // user must provide the stand alone profiler
       StandAloneProfiler profiler = Util.getArg(1, StandAloneProfiler.class, args);
       try {
-        profiler.apply(message);
+        for(JSONObject message : messages) {
+          profiler.apply(message);
+        }
 
       } catch(ExecutionException e) {
-        throw new IllegalArgumentException(e);
+        throw new IllegalArgumentException(format("Failed to apply message; error=%s", e.getMessage()),
e);
       }
 
       return profiler;

http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java
b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java
index 9cc8046..bad3efe 100644
--- a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java
+++ b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/ProfilerFunctionsTest.java
@@ -58,6 +58,28 @@ public class ProfilerFunctionsTest {
   private String message;
 
   /**
+   * [
+   * {
+   *    "ip_src_addr": "10.0.0.1",
+   *    "ip_dst_addr": "10.0.0.2",
+   *    "source.type": "test",
+   * },
+   * {
+   *    "ip_src_addr": "10.0.0.1",
+   *    "ip_dst_addr": "10.0.0.2",
+   *    "source.type": "test",
+   * },
+   * {
+   *    "ip_src_addr": "10.0.0.1",
+   *    "ip_dst_addr": "10.0.0.2",
+   *    "source.type": "test",
+   * }
+   * ]
+   */
+  @Multiline
+  private String messages;
+
+  /**
    * {
    *   "profiles": [
    *        {
@@ -108,7 +130,9 @@ public class ProfilerFunctionsTest {
     state.put("config", "{ \"profiles\" : [] }");
     StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class);
     assertNotNull(profiler);
-    assertEquals(0, profiler.getConfig().getProfiles().size());
+    assertEquals(0, profiler.getProfileCount());
+    assertEquals(0, profiler.getMessageCount());
+    assertEquals(0, profiler.getRouteCount());
   }
 
   @Test
@@ -116,7 +140,9 @@ public class ProfilerFunctionsTest {
     state.put("config", helloWorldProfilerDef);
     StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class);
     assertNotNull(profiler);
-    assertEquals(1, profiler.getConfig().getProfiles().size());
+    assertEquals(1, profiler.getProfileCount());
+    assertEquals(0, profiler.getMessageCount());
+    assertEquals(0, profiler.getRouteCount());
   }
 
   @Test(expected = IllegalArgumentException.class)
@@ -144,7 +170,9 @@ public class ProfilerFunctionsTest {
     StandAloneProfiler profiler = executor.execute(expression, state, StandAloneProfiler.class);
 
     assertNotNull(profiler);
-    assertEquals(1, profiler.getConfig().getProfiles().size());
+    assertEquals(1, profiler.getProfileCount());
+    assertEquals(0, profiler.getMessageCount());
+    assertEquals(0, profiler.getRouteCount());
   }
 
   @Test
@@ -158,19 +186,74 @@ public class ProfilerFunctionsTest {
     // apply a message to the profiler
     state.put("message", message);
     StandAloneProfiler result = run("PROFILER_APPLY(message, profiler)", StandAloneProfiler.class);
+
+    // validate
+    assertSame(profiler, result);
+    assertEquals(1, profiler.getProfileCount());
+    assertEquals(1, profiler.getMessageCount());
+    assertEquals(1, profiler.getRouteCount());
+  }
+
+  @Test
+  public void testProfilerApplyWithMultipleMessages() {
+
+    // initialize the profiler
+    state.put("config", helloWorldProfilerDef);
+    StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class);
+    state.put("profiler", profiler);
+
+    // apply a message to the profiler
+    state.put("messages", messages);
+    StandAloneProfiler result = run("PROFILER_APPLY(messages, profiler)", StandAloneProfiler.class);
+
+    // validate
     assertSame(profiler, result);
+    assertEquals(1, profiler.getProfileCount());
+    assertEquals(3, profiler.getMessageCount());
+    assertEquals(3, profiler.getRouteCount());
+  }
+
+  @Test
+  public void testProfilerApplyWithEmptyList() {
+
+    // initialize the profiler
+    state.put("config", helloWorldProfilerDef);
+    StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class);
+    state.put("profiler", profiler);
+
+    // apply a message to the profiler
+    state.put("messages", "[ ]");
+    StandAloneProfiler result = run("PROFILER_APPLY(messages, profiler)", StandAloneProfiler.class);
+
+    // validate
+    assertSame(profiler, result);
+    assertEquals(1, profiler.getProfileCount());
+    assertEquals(0, profiler.getMessageCount());
+    assertEquals(0, profiler.getRouteCount());
   }
 
   @Test(expected = IllegalArgumentException.class)
-  public void testProfilerApplyNoArgs() {
+  public void testProfilerApplyWithNoArgs() {
     run("PROFILER_APPLY()", StandAloneProfiler.class);
   }
 
   @Test(expected = IllegalArgumentException.class)
-  public void testProfilerApplyInvalidArg() {
+  public void testProfilerApplyWithInvalidArg() {
     run("PROFILER_APPLY(undefined)", StandAloneProfiler.class);
   }
 
+  @Test(expected = IllegalArgumentException.class)
+  public void testProfilerApplyWithNullMessage() {
+
+    // initialize the profiler
+    state.put("config", helloWorldProfilerDef);
+    StandAloneProfiler profiler = run("PROFILER_INIT(config)", StandAloneProfiler.class);
+    state.put("profiler", profiler);
+
+    // there is no 'messages' variable
+    StandAloneProfiler result = run("PROFILER_APPLY(messages, profiler)", StandAloneProfiler.class);
+  }
+
   @Test
   public void testProfilerFlush() {
 

http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java
b/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java
index cf034c8..6db7079 100644
--- a/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java
+++ b/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/StandAloneProfiler.java
@@ -53,12 +53,28 @@ public class StandAloneProfiler {
    */
   private MessageDistributor distributor;
 
+  /**
+   * Counts the number of messages that have been applied.
+   */
+  private int messageCount;
+
+  /**
+   * Counts the number of routes.
+   *
+   * If a message is not needed by any profiles, then there are 0 routes.
+   * If a message is needed by 1 profile then there is 1 route.
+   * If a message is needed by 2 profiles then there are 2 routes.
+   */
+  private int routeCount;
+
   public StandAloneProfiler(ProfilerConfig config, long periodDurationMillis, Context context)
{
     this.context = context;
     this.config = config;
     this.router = new DefaultMessageRouter(context);
     // the period TTL does not matter in this context
     this.distributor = new DefaultMessageDistributor(periodDurationMillis, Long.MAX_VALUE);
+    this.messageCount = 0;
+    this.routeCount = 0;
   }
 
   /**
@@ -72,6 +88,18 @@ public class StandAloneProfiler {
     for(MessageRoute route : routes) {
       distributor.distribute(message, route, context);
     }
+
+    routeCount += routes.size();
+    messageCount += 1;
+  }
+
+  @Override
+  public String toString() {
+    return "Profiler{" +
+            getProfileCount() + " profile(s), " +
+            getMessageCount() + " messages(s), " +
+            getRouteCount() + " route(s)" +
+            '}';
   }
 
   /**
@@ -85,4 +113,16 @@ public class StandAloneProfiler {
   public ProfilerConfig getConfig() {
     return config;
   }
+
+  public int getProfileCount() {
+    return (config == null) ? 0: config.getProfiles().size();
+  }
+
+  public int getMessageCount() {
+    return messageCount;
+  }
+
+  public int getRouteCount() {
+    return routeCount;
+  }
 }

http://git-wip-us.apache.org/repos/asf/metron/blob/c73c7af1/metron-analytics/metron-profiler/README.md
----------------------------------------------------------------------
diff --git a/metron-analytics/metron-profiler/README.md b/metron-analytics/metron-profiler/README.md
index f27af59..b7aed55 100644
--- a/metron-analytics/metron-profiler/README.md
+++ b/metron-analytics/metron-profiler/README.md
@@ -281,6 +281,8 @@ In the following example, three values, the minimum, the maximum and the
mean ar
 
 A numeric value that defines how many days the profile data is retained.  After this time,
the data expires and is no longer accessible.  If no value is defined, the data does not expire.
 
+The REPL can be a powerful for developing profiles. Read all about [Developing Profiles](../metron-profiler-client/#developing_profiles).
+
 ## Configuring the Profiler
 
 The Profiler runs as an independent Storm topology.  The configuration for the Profiler topology
is stored in local filesystem at `$METRON_HOME/config/profiler.properties`.


Mime
View raw message