metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: Profiler issue
Date Fri, 24 Feb 2017 15:02:46 GMT
Yeah, we snapshot every 15 minutes.  You can adjust that if you wnat to
snapshot the data in the profiler to hbase quicker.  Glad your issue went
away :)

On Fri, Feb 24, 2017 at 10:00 AM, shoggi <shoggi@gmail.com> wrote:

> Hi
>
> The error has gone and I get now data into the profiler (base.png). The
> time delay is large though. As a note, the domain_without_subdomains is in
> my example never empty (es.png).
>
> When I try to query the data in stellar, then I get an error but only if I
> the query matches data.
>
> Thank you
> shoggi
>
>
> On Fri, Feb 24, 2017 at 3:25 PM, Casey Stella <cestella@gmail.com> wrote:
>
>> I wonder if one of your records has a null domain_without_subdomains or
>> it doesn't exist for one of them.  Try the following config:
>> {
>>   "profiles": [
>>     {
>>       "profile": "url-length",
>>       "foreach": "if exists(domain_without_subdomains) then
>> domain_without_subdomains else 'n/a'",
>>       "onlyif": "exists(domain_without_subdomains) && source.type ==
>> 'squid'",
>>       "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>>       "result": "m"
>>     }
>>   ]
>> }
>>
>>
>> On Fri, Feb 24, 2017 at 9:09 AM, shoggi <shoggi@gmail.com> wrote:
>>
>>> Sure
>>>
>>> Here is the output..
>>>
>>> # ./metron-deployment/scripts/platform-info.sh
>>> Metron 0.3.0
>>> --
>>> * master
>>> --
>>> commit 59f8f03acbf66d5ac63f9156546833b0c98f57f2
>>> Author: mmiklavc <michael.miklavcic@gmail.com>
>>> Date:   Wed Nov 30 11:24:05 2016 -0700
>>>
>>>     METRON-565: apps/metron/enrichment/indexed directory path does not
>>> get created for metron cluster deployed via Ambari (mmiklavc) closes
>>> apache/incubator-metron#365
>>> --
>>>  .../roles/elasticsearch/tasks/elasticsearch.yml    |    9 ++++-----
>>>  .../roles/pcap_replay/templates/pcap-replay        |    9 ++++++---
>>>  2 files changed, 10 insertions(+), 8 deletions(-)
>>> --
>>> ./metron-deployment/scripts/platform-info.sh: line 48: ansible: command
>>> not found
>>> --
>>> ./metron-deployment/scripts/platform-info.sh: line 52: vagrant: command
>>> not found
>>> --
>>> Python 2.6.6
>>> --
>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>> 2015-11-10T17:41:47+01:00)
>>> Maven home: /opt/apache-maven-3.3.9
>>> Java version: 1.8.0_111, vendor: Oracle Corporation
>>> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_64
>>> /jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "linux", version: "2.6.32-642.11.1.el6.x86_64", arch: "amd64",
>>> family: "unix"
>>> --
>>> Linux x 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC 2016
>>> x86_64 x86_64 x86_64 GNU/Linux
>>>
>>>
>>>
>>> On Fri, Feb 24, 2017 at 2:55 PM, Nick Allen <nick@nickallen.org> wrote:
>>>
>>>> Can you run the "./metron-deployment/scripts/platform-info.sh" script
>>>> and send the output to us?
>>>>
>>>> On Fri, Feb 24, 2017 at 8:38 AM, shoggi <shoggi@gmail.com> wrote:
>>>>
>>>>> Hi All
>>>>>
>>>>> I tried to set up following profiler config.
>>>>>
>>>>> > CONFIG_GET("PROFILER")
>>>>> {
>>>>>   "profiles": [
>>>>>     {
>>>>>       "profile": "url-length",
>>>>>       "foreach": "domain_without_subdomains",
>>>>>       "onlyif": "source.type == 'squid'",
>>>>>       "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>>>>>       "result": "m"
>>>>>     }
>>>>>   ]
>>>>> }
>>>>>
>>>>> Whenever I send squid data through, I get a null pointer exception &
>>>>> the profiler dies. Other enrichments on the squid feed work fine and
the
>>>>> enriched logs reaches elasticsearch. I followed this guide:
>>>>> https://cwiki.apache.org/confluence/display/METRON/Pr
>>>>> ofiling+Your+Streams to check my syntax and proceedings.
>>>>>
>>>>> Any help is highly appreciated.
>>>>>
>>>>> Cheers
>>>>> Shoji
>>>>>
>>>>>
>>>>>
>>>>> 2017-02-24 14:20:35.832 o.a.s.util [ERROR] Async loop died!
>>>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>>>         at org.apache.storm.utils.Disrupt
>>>>> orQueue.consumeBatchToCursor(DisruptorQueue.java:464)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.utils.Disrupt
>>>>> orQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.disruptor$con
>>>>> sume_batch_when_available.invoke(disruptor.clj:73)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.daemon.execut
>>>>> or$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.util$async_lo
>>>>> op$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.0.0-1245
>>>>> .jar:1.0.1.2.5.0.0-1245]
>>>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>> Caused by: java.lang.NullPointerException
>>>>>         at org.apache.metron.profiler.ste
>>>>> llar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:117)
>>>>> ~[stormjar.jar:?]
>>>>>         at org.apache.metron.profiler.bol
>>>>> t.ProfileSplitterBolt.applyProfile(ProfileSplitterBolt.java:139)
>>>>> ~[stormjar.jar:?]
>>>>>         at org.apache.metron.profiler.bol
>>>>> t.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:122)
>>>>> ~[stormjar.jar:?]
>>>>>         at org.apache.metron.profiler.bol
>>>>> t.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:97)
>>>>> ~[stormjar.jar:?]
>>>>>         at org.apache.storm.daemon.execut
>>>>> or$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.daemon.execut
>>>>> or$mk_task_receiver$fn__6492.invoke(executor.clj:466)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.disruptor$clo
>>>>> jure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.utils.Disrupt
>>>>> orQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         ... 6 more
>>>>> 2017-02-24 14:20:35.834 o.a.s.d.executor [ERROR]
>>>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>>>         at org.apache.storm.utils.Disrupt
>>>>> orQueue.consumeBatchToCursor(DisruptorQueue.java:464)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.utils.Disrupt
>>>>> orQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.disruptor$con
>>>>> sume_batch_when_available.invoke(disruptor.clj:73)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.daemon.execut
>>>>> or$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.util$async_lo
>>>>> op$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.0.0-1245
>>>>> .jar:1.0.1.2.5.0.0-1245]
>>>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>> Caused by: java.lang.NullPointerException
>>>>>         at org.apache.metron.profiler.ste
>>>>> llar.DefaultStellarExecutor.execute(DefaultStellarExecutor.java:117)
>>>>> ~[stormjar.jar:?]
>>>>>         at org.apache.metron.profiler.bol
>>>>> t.ProfileSplitterBolt.applyProfile(ProfileSplitterBolt.java:139)
>>>>> ~[stormjar.jar:?]
>>>>>         at org.apache.metron.profiler.bol
>>>>> t.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:122)
>>>>> ~[stormjar.jar:?]
>>>>>         at org.apache.metron.profiler.bol
>>>>> t.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:97)
>>>>> ~[stormjar.jar:?]
>>>>>         at org.apache.storm.daemon.execut
>>>>> or$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.daemon.execut
>>>>> or$mk_task_receiver$fn__6492.invoke(executor.clj:466)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.disruptor$clo
>>>>> jure_handler$reify__6005.onEvent(disruptor.clj:40)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.utils.Disrupt
>>>>> orQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         ... 6 more
>>>>> 2017-02-24 14:20:35.908 o.a.s.util [ERROR] Halting process: ("Worker
>>>>> died")
>>>>> java.lang.RuntimeException: ("Worker died")
>>>>>         at org.apache.storm.util$exit_pro
>>>>> cess_BANG_.doInvoke(util.clj:341) [storm-core-1.0.1.2.5.0.0-1245
>>>>> .jar:1.0.1.2.5.0.0-1245]
>>>>>         at clojure.lang.RestFn.invoke(RestFn.java:423)
>>>>> [clojure-1.7.0.jar:?]
>>>>>         at org.apache.storm.daemon.worker
>>>>> $fn__7176$fn__7177.invoke(worker.clj:765)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.daemon.execut
>>>>> or$mk_executor_data$fn__6388$fn__6389.invoke(executor.clj:275)
>>>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>>>         at org.apache.storm.util$async_lo
>>>>> op$fn__554.invoke(util.clj:494) [storm-core-1.0.1.2.5.0.0-1245
>>>>> .jar:1.0.1.2.5.0.0-1245]
>>>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message