metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: Profiler issue
Date Sat, 25 Feb 2017 01:07:26 GMT
What's the error exactly on 2?
On Fri, Feb 24, 2017 at 19:36 shoggi <shoggi@gmail.com> wrote:

> Sorry for my late reply. From my previous post I still have two open
> questions.
>
> 1) My domain_without_subdomains entries are not empty, why would I need to
> add the 'if exists' statement? I'd like to understand the logic behind that.
>
> 2) I get now data slowly into the profiler (even with 1 minute profiler
> pushes). But I do get Java errors in Stellar on any queries based on that
> profiler data. For example running "PROFILE_GET('url-length', '20min.ch',
> 15, 'MINUTES')", throws an error. I can adjust the durationAgo were no data
> is available and the command returns empty. I included a screenshot in my
> previous mail.
>
> Do you have any insight into that?
>
> On Fri, Feb 24, 2017 at 4:02 PM, Casey Stella <cestella@gmail.com> wrote:
>
> Yeah, we snapshot every 15 minutes.  You can adjust that if you wnat to
> snapshot the data in the profiler to hbase quicker.  Glad your issue went
> away :)
>
> On Fri, Feb 24, 2017 at 10:00 AM, shoggi <shoggi@gmail.com> wrote:
>
> Hi
>
> The error has gone and I get now data into the profiler (base.png). The
> time delay is large though. As a note, the domain_without_subdomains is in
> my example never empty (es.png).
>
> When I try to query the data in stellar, then I get an error but only if I
> the query matches data.
>
> Thank you
> shoggi
>
>
> On Fri, Feb 24, 2017 at 3:25 PM, Casey Stella <cestella@gmail.com> wrote:
>
> I wonder if one of your records has a null domain_without_subdomains or it
> doesn't exist for one of them.  Try the following config:
> {
>   "profiles": [
>     {
>       "profile": "url-length",
>       "foreach": "if exists(domain_without_subdomains) then
> domain_without_subdomains else 'n/a'",
>       "onlyif": "exists(domain_without_subdomains) && source.type ==
> 'squid'",
>       "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>       "result": "m"
>     }
>   ]
> }
>
>
> On Fri, Feb 24, 2017 at 9:09 AM, shoggi <shoggi@gmail.com> wrote:
>
> Sure
>
> Here is the output..
>
> # ./metron-deployment/scripts/platform-info.sh
> Metron 0.3.0
> --
> * master
> --
> commit 59f8f03acbf66d5ac63f9156546833b0c98f57f2
> Author: mmiklavc <michael.miklavcic@gmail.com>
> Date:   Wed Nov 30 11:24:05 2016 -0700
>
>     METRON-565: apps/metron/enrichment/indexed directory path does not get
> created for metron cluster deployed via Ambari (mmiklavc) closes
> apache/incubator-metron#365
> --
>  .../roles/elasticsearch/tasks/elasticsearch.yml    |    9 ++++-----
>  .../roles/pcap_replay/templates/pcap-replay        |    9 ++++++---
>  2 files changed, 10 insertions(+), 8 deletions(-)
> --
> ./metron-deployment/scripts/platform-info.sh: line 48: ansible: command
> not found
> --
> ./metron-deployment/scripts/platform-info.sh: line 52: vagrant: command
> not found
> --
> Python 2.6.6
> --
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T17:41:47+01:00)
> Maven home: /opt/apache-maven-3.3.9
> Java version: 1.8.0_111, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "2.6.32-642.11.1.el6.x86_64", arch: "amd64",
> family: "unix"
> --
> Linux x 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC 2016
> x86_64 x86_64 x86_64 GNU/Linux
>
>
>
> On Fri, Feb 24, 2017 at 2:55 PM, Nick Allen <nick@nickallen.org> wrote:
>
> Can you run the "./metron-deployment/scripts/platform-info.sh" script and
> send the output to us?
>
> On Fri, Feb 24, 2017 at 8:38 AM, shoggi <shoggi@gmail.com> wrote:
>
> Hi All
>
> I tried to set up following profiler config.
>
> > CONFIG_GET("PROFILER")
> {
>   "profiles": [
>     {
>       "profile": "url-length",
>       "foreach": "domain_without_subdomains",
>       "onlyif": "source.type == 'squid'",
>       "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>       "result": "m"
>     }
>   ]
> }
>
> Whenever I send squid data through, I get a null pointer exception & the
> profiler dies. Other enrichments on the squid feed work fine and the
> enriched logs reaches elasticsearch. I followed this guide:
> https://cwiki.apache.org/confluence/display/METRON/Profiling+Your+Streams to
> check my syntax and proceedings.
>
> Any help is highly appreciated.
>
> Cheers
> Shoji
>
>
>
> 2017-02-24 14:20:35.832 o.a.s.util [ERROR] Async loop died!
> java.lang.RuntimeException: java.lang.NullPointerException
>         at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
>         at
> org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:117)
> ~[stormjar.jar:?]
>         at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.applyProfile(ProfileSplitterBolt.java:139)
> ~[stormjar.jar:?]
>         at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:122)
> ~[stormjar.jar:?]
>         at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:97)
> ~[stormjar.jar:?]
>         at
> org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         ... 6 more
> 2017-02-24 14:20:35.834 o.a.s.d.executor [ERROR]
> java.lang.RuntimeException: java.lang.NullPointerException
>         at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.NullPointerException
>         at
> org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:117)
> ~[stormjar.jar:?]
>         at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.applyProfile(ProfileSplitterBolt.java:139)
> ~[stormjar.jar:?]
>         at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:122)
> ~[stormjar.jar:?]
>         at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:97)
> ~[stormjar.jar:?]
>         at
> org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         ... 6 more
> 2017-02-24 14:20:35.908 o.a.s.util [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
>         at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at clojure.lang.RestFn.invoke(RestFn.java:423)
> [clojure-1.7.0.jar:?]
>         at
> org.apache.storm.daemon.worker$fn__7176$fn__7177.invoke(worker.clj:765)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at
> org.apache.storm.daemon.executor$mk_executor_data$fn__6388$fn__6389.invoke(executor.clj:275)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:494)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>
>
>
>
>
>
>
>
>
>

Mime
View raw message