metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shoggi <sho...@gmail.com>
Subject Re: Profiler issue
Date Sat, 25 Feb 2017 01:50:15 GMT
I managed to get it to run. The lib filter is too strict and the main
stellar class will not load but having this in a separate playground made
it work. Unfortunately I cannot upgrade but hopefully the stats function
will work in the enrichment part and then this does not bother me too much.

Thank you Casey !

On Sat, Feb 25, 2017 at 2:12 AM, shoggi <shoggi@gmail.com> wrote:

> If I select a durationAgo value which does not contain any profiler data I
> get this, which is normal:
> [Stellar]>>> PROFILE_GET('url-length', '20min.ch', 15, 'MINUTES')
> []
>
> If I select a value which has profiler data available, then I always get
> an exception. I can repeat that with any entity stored in the profiler.
>
> [Stellar]>>> PROFILE_GET('url-length', '20min.ch', 120, 'MINUTES')
> [!] Unable to execute: com.tdunning.math.stats.
> TDigest.createAvlTreeDigest(D)Lcom/tdunning/math/stats/TDigest;
> org.apache.metron.common.dsl.ParseException: Unable to execute:
> com.tdunning.math.stats.TDigest.createAvlTreeDigest(D)
> Lcom/tdunning/math/stats/TDigest;
> at org.apache.metron.common.stellar.StellarCompiler.
> getResult(StellarCompiler.java:428)
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:90)
> at org.apache.metron.common.stellar.shell.StellarExecutor.
> execute(StellarExecutor.java:275)
> at org.apache.metron.common.stellar.shell.StellarShell.
> executeStellar(StellarShell.java:373)
> at org.apache.metron.common.stellar.shell.StellarShell.
> handleStellar(StellarShell.java:276)
> at org.apache.metron.common.stellar.shell.StellarShell.
> execute(StellarShell.java:412)
> at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoSuchMethodError: com.tdunning.math.stats.
> TDigest.createAvlTreeDigest(D)Lcom/tdunning/math/stats/TDigest;
> at org.apache.metron.statistics.OnlineStatisticsProvider.<init>(
> OnlineStatisticsProvider.java:74)
> at org.apache.metron.statistics.OnlineStatisticsProviderConstr
> uctorAccess.newInstance(Unknown Source)
> at org.apache.metron.common.utils.SerDeUtils$
> DefaultInstantiatorStrategy$1.newInstance(SerDeUtils.java:132)
> at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1090)
> at com.esotericsoftware.kryo.serializers.DefaultSerializers$
> KryoSerializableSerializer.read(DefaultSerializers.java:509)
> at com.esotericsoftware.kryo.serializers.DefaultSerializers$
> KryoSerializableSerializer.read(DefaultSerializers.java:503)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790)
> at org.apache.metron.common.utils.SerDeUtils.fromBytes(
> SerDeUtils.java:246)
> at org.apache.metron.profiler.client.HBaseProfilerClient.lambda$get$3(
> HBaseProfilerClient.java:130)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.ReferencePipeline$3$1.accept(
> ReferencePipeline.java:193)
> at java.util.stream.ReferencePipeline$2$1.accept(
> ReferencePipeline.java:175)
> at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.
> java:948)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(
> AbstractPipeline.java:471)
> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(
> ForEachOps.java:151)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.
> evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
> at org.apache.metron.profiler.client.HBaseProfilerClient.
> get(HBaseProfilerClient.java:130)
> at org.apache.metron.profiler.client.HBaseProfilerClient.
> fetch(HBaseProfilerClient.java:109)
> at org.apache.metron.profiler.client.HBaseProfilerClient.
> fetch(HBaseProfilerClient.java:80)
> at org.apache.metron.profiler.client.stellar.GetProfile.
> apply(GetProfile.java:181)
> at org.apache.metron.common.stellar.StellarCompiler.
> exitTransformationFunc(StellarCompiler.java:246)
> at org.apache.metron.common.stellar.generated.StellarParser$
> TransformationFuncContext.exitRule(StellarParser.java:1617)
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
> at org.apache.metron.common.stellar.generated.StellarParser.functions(
> StellarParser.java:1640)
> at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_
> operands(StellarParser.java:1750)
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr(StellarParser.java:1397)
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation_expr(StellarParser.java:278)
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation(StellarParser.java:147)
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:89)
> ... 8 more
>
> On Sat, Feb 25, 2017 at 2:07 AM, Casey Stella <cestella@gmail.com> wrote:
>
>> What's the error exactly on 2?
>>
>> On Fri, Feb 24, 2017 at 19:36 shoggi <shoggi@gmail.com> wrote:
>>
>>> Sorry for my late reply. From my previous post I still have two open
>>> questions.
>>>
>>> 1) My domain_without_subdomains entries are not empty, why would I need
>>> to add the 'if exists' statement? I'd like to understand the logic behind
>>> that.
>>>
>>> 2) I get now data slowly into the profiler (even with 1 minute profiler
>>> pushes). But I do get Java errors in Stellar on any queries based on that
>>> profiler data. For example running "PROFILE_GET('url-length', '20min.ch',
>>> 15, 'MINUTES')", throws an error. I can adjust the durationAgo were no data
>>> is available and the command returns empty. I included a screenshot in my
>>> previous mail.
>>>
>>> Do you have any insight into that?
>>>
>>> On Fri, Feb 24, 2017 at 4:02 PM, Casey Stella <cestella@gmail.com>
>>> wrote:
>>>
>>> Yeah, we snapshot every 15 minutes.  You can adjust that if you wnat to
>>> snapshot the data in the profiler to hbase quicker.  Glad your issue went
>>> away :)
>>>
>>> On Fri, Feb 24, 2017 at 10:00 AM, shoggi <shoggi@gmail.com> wrote:
>>>
>>> Hi
>>>
>>> The error has gone and I get now data into the profiler (base.png). The
>>> time delay is large though. As a note, the domain_without_subdomains is in
>>> my example never empty (es.png).
>>>
>>> When I try to query the data in stellar, then I get an error but only if
>>> I the query matches data.
>>>
>>> Thank you
>>> shoggi
>>>
>>>
>>> On Fri, Feb 24, 2017 at 3:25 PM, Casey Stella <cestella@gmail.com>
>>> wrote:
>>>
>>> I wonder if one of your records has a null domain_without_subdomains or
>>> it doesn't exist for one of them.  Try the following config:
>>> {
>>>   "profiles": [
>>>     {
>>>       "profile": "url-length",
>>>       "foreach": "if exists(domain_without_subdomains) then
>>> domain_without_subdomains else 'n/a'",
>>>       "onlyif": "exists(domain_without_subdomains) && source.type ==
>>> 'squid'",
>>>       "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>>>       "result": "m"
>>>     }
>>>   ]
>>> }
>>>
>>>
>>> On Fri, Feb 24, 2017 at 9:09 AM, shoggi <shoggi@gmail.com> wrote:
>>>
>>> Sure
>>>
>>> Here is the output..
>>>
>>> # ./metron-deployment/scripts/platform-info.sh
>>> Metron 0.3.0
>>> --
>>> * master
>>> --
>>> commit 59f8f03acbf66d5ac63f9156546833b0c98f57f2
>>> Author: mmiklavc <michael.miklavcic@gmail.com>
>>> Date:   Wed Nov 30 11:24:05 2016 -0700
>>>
>>>     METRON-565: apps/metron/enrichment/indexed directory path does not
>>> get created for metron cluster deployed via Ambari (mmiklavc) closes
>>> apache/incubator-metron#365
>>> --
>>>  .../roles/elasticsearch/tasks/elasticsearch.yml    |    9 ++++-----
>>>  .../roles/pcap_replay/templates/pcap-replay        |    9 ++++++---
>>>  2 files changed, 10 insertions(+), 8 deletions(-)
>>> --
>>> ./metron-deployment/scripts/platform-info.sh: line 48: ansible: command
>>> not found
>>> --
>>> ./metron-deployment/scripts/platform-info.sh: line 52: vagrant: command
>>> not found
>>> --
>>> Python 2.6.6
>>> --
>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5;
>>> 2015-11-10T17:41:47+01:00)
>>> Maven home: /opt/apache-maven-3.3.9
>>> Java version: 1.8.0_111, vendor: Oracle Corporation
>>> Java home: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-0.b15.el6_8.x86_
>>> 64/jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "linux", version: "2.6.32-642.11.1.el6.x86_64", arch: "amd64",
>>> family: "unix"
>>> --
>>> Linux x 2.6.32-642.11.1.el6.x86_64 #1 SMP Fri Nov 18 19:25:05 UTC 2016
>>> x86_64 x86_64 x86_64 GNU/Linux
>>>
>>>
>>>
>>> On Fri, Feb 24, 2017 at 2:55 PM, Nick Allen <nick@nickallen.org> wrote:
>>>
>>> Can you run the "./metron-deployment/scripts/platform-info.sh" script
>>> and send the output to us?
>>>
>>> On Fri, Feb 24, 2017 at 8:38 AM, shoggi <shoggi@gmail.com> wrote:
>>>
>>> Hi All
>>>
>>> I tried to set up following profiler config.
>>>
>>> > CONFIG_GET("PROFILER")
>>> {
>>>   "profiles": [
>>>     {
>>>       "profile": "url-length",
>>>       "foreach": "domain_without_subdomains",
>>>       "onlyif": "source.type == 'squid'",
>>>       "update": { "m": "STATS_ADD(m, LENGTH(url))" },
>>>       "result": "m"
>>>     }
>>>   ]
>>> }
>>>
>>> Whenever I send squid data through, I get a null pointer exception & the
>>> profiler dies. Other enrichments on the squid feed work fine and the
>>> enriched logs reaches elasticsearch. I followed this guide:
>>> https://cwiki.apache.org/confluence/display/METRON/Pr
>>> ofiling+Your+Streams to check my syntax and proceedings.
>>>
>>> Any help is highly appreciated.
>>>
>>> Cheers
>>> Shoji
>>>
>>>
>>>
>>> 2017-02-24 14:20:35.832 o.a.s.util [ERROR] Async loop died!
>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.NullPointerException
>>>         at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:117) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.applyPro
>>> file(ProfileSplitterBolt.java:139) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecut
>>> e(ProfileSplitterBolt.java:122) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:97)
>>> ~[stormjar.jar:?]
>>>         at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         ... 6 more
>>> 2017-02-24 14:20:35.834 o.a.s.d.executor [ERROR]
>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.NullPointerException
>>>         at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:117) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.applyPro
>>> file(ProfileSplitterBolt.java:139) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecut
>>> e(ProfileSplitterBolt.java:122) ~[stormjar.jar:?]
>>>         at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:97)
>>> ~[stormjar.jar:?]
>>>         at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:466)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> ~[storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         ... 6 more
>>> 2017-02-24 14:20:35.908 o.a.s.util [ERROR] Halting process: ("Worker
>>> died")
>>> java.lang.RuntimeException: ("Worker died")
>>>         at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at clojure.lang.RestFn.invoke(RestFn.java:423)
>>> [clojure-1.7.0.jar:?]
>>>         at org.apache.storm.daemon.worker$fn__7176$fn__7177.invoke(worker.clj:765)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.daemon.executor$mk_executor_data$fn__6388$
>>> fn__6389.invoke(executor.clj:275) [storm-core-1.0.1.2.5.0.0-1245
>>> .jar:1.0.1.2.5.0.0-1245]
>>>         at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:494)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>

Mime
View raw message