metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: Profiler issue
Date Fri, 24 Feb 2017 14:25:39 GMT
I wonder if one of your records has a null domain_without_subdomains or it
doesn't exist for one of them.  Try the following config:
{
  "profiles": [
    {
      "profile": "url-length",
      "foreach": "if exists(domain_without_subdomains) then
domain_without_subdomains else 'n/a'",
      "onlyif": "exists(domain_without_subdomains) && source.type ==
'squid'",
      "update": { "m": "STATS_ADD(m, LENGTH(url))" },
      "result": "m"
    }
  ]
}


On Fri, Feb 24, 2017 at 9:09 AM, shoggi <shoggi@gmail.com> wrote:

> Sure
>
> Here is the output..
>
> # ./metron-deployment/scripts/platform-info.sh
> Metron 0.3.0
> --
> * master
> --
> commit 59f8f03acbf66d5ac63f9156546833b0c98f57f2
> Author: mmiklavc <michael.miklavcic@gmail.com>
> Date:   Wed Nov 30 11:24:05 2016 -0700
>
>     METRON-565: apps/metron/enrichment/indexed directory path does not get
> created for metron cluster deployed via Ambari (mmiklavc) closes
> apache/incubator-metron#365
> --
>  .../roles/elasticsearch/tasks/elasticsearch.yml    |    9 ++++-----
>  .../roles/pcap_replay/templates/pcap-replay        |    9 ++++++---
>  2 files changed, 10 insertions(+), 8 deletions(-)
> --
> ./metron-deployment/scripts/platform-info.sh: line 48: ansible: command
> not found
> --
> ./metron-deployment/scripts/platform-info.sh: line 52: vagrant: command
> not found
> --
> Python 2.6.6
> --
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
> 2015-11-10T17:41:47+01:00)
> Maven home: /opt/apache-maven-3.3.9
> Java version: 1.8.0_111, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.
> x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "2.6.32-642.11.1.el6.x86_64", arch: "amd64",
> family: "unix"
> --
> Linux x 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC 2016
> x86_64 x86_64 x86_64 GNU/Linux
>
>
>
> On Fri, Feb 24, 2017 at 2:55 PM, Nick Allen <nick@nickallen.org> wrote:
>
>> Can you run the "./metron-deployment/scripts/platform-info.sh" script
>> and send the output to us?
>>
>> On Fri, Feb 24, 2017 at 8:38 AM, shoggi <shoggi@gmail.com> wrote:
>>
>>> Hi All
>>>
>>> I tried to set up following profiler config.
>>>
>>> > CONFIG_GET("PROFILER")
>>> {
>>>   "profiles": [
>>>     {
>>>       "profile": "url-length",
>>>       "foreach": "domain_without_subdomains",
>>>       "onlyif": "source.type == 'squid'",
>>>       "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>>>       "result": "m"
>>>     }
>>>   ]
>>> }
>>>
>>> Whenever I send squid data through, I get a null pointer exception & the
>>> profiler dies. Other enrichments on the squid feed work fine and the
>>> enriched logs reaches elasticsearch. I followed this guide:
>>> https://cwiki.apache.org/confluence/display/METRON/Pr
>>> ofiling+Your+Streams to check my syntax and proceedings.
>>>
>>> Any help is highly appreciated.
>>>
>>> Cheers
>>> Shoji
>>>
>>>
>>>
>>> 2017-02-24 14:20:35.832 o.a.s.util [ERROR] Async loop died!
>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.NullPointerException
>>>         at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:117) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.applyPro
>>> file(ProfileSplitterBolt.java:139) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecut
>>> e(ProfileSplitterBolt.java:122) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:97)
>>> ~[stormjar.jar:?]
>>>         at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         ... 6 more
>>> 2017-02-24 14:20:35.834 o.a.s.d.executor [ERROR]
>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.NullPointerException
>>>         at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:117) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.applyPro
>>> file(ProfileSplitterBolt.java:139) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecut
>>> e(ProfileSplitterBolt.java:122) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:97)
>>> ~[stormjar.jar:?]
>>>         at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         ... 6 more
>>> 2017-02-24 14:20:35.908 o.a.s.util [ERROR] Halting process: ("Worker
>>> died")
>>> java.lang.RuntimeException: ("Worker died")
>>>         at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at clojure.lang.RestFn.invoke(RestFn.java:423)
>>> [clojure-1.7.0.jar:?]
>>>         at org.apache.storm.daemon.worker$fn__7176$fn__7177.invoke(worker.clj:765)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$mk_executor_data$fn__6388$f
>>> n__6389.invoke(executor.clj:275) [storm-core-1.0.1.2.5.0.0-1245
>>> .jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:494)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>
>>>
>>>
>>>
>>
>

Mime
View raw message