metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: Median Absolute Deviation - Profiler Error
Date Mon, 27 Feb 2017 23:18:09 GMT
Ok, try something for me and tell me if it fixes it: adjust the config to
remove the tickUpdate (it didn't work anyway and we have since removed it
in 0.3.1).


On Mon, Feb 27, 2017 at 6:07 PM, shoggi <shoggi@gmail.com> wrote:

> Yes, this is 0.3.0.
>
> Here is the full trace.
>
> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
> profile: profile=sketchy_mad, entity=krxd.net
> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
> failure: message='Unable to execute: org.apache.metron.statistics.outlier.
> MedianAbsoluteDeviationFunctions$State cannot be cast to java.util.List',
> tuple='source: __system:-1, stream: __tick, id: {}, [900]'
> org.apache.metron.common.dsl.ParseException: Unable to execute:
> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
> cannot be cast to java.util.List
> at org.apache.metron.common.stellar.StellarCompiler.
> getResult(StellarCompiler.java:428) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:90) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(
> DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(
> DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
> ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> doExecute(ProfileBuilderBolt.java:164) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> execute(ProfileBuilderBolt.java:144) [stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.
> outlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
> java.util.List
> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctio
> ns$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.StellarCompiler.
> exitTransformationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser$
> TransformationFuncContext.exitRule(StellarParser.java:1617)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_
> operands(StellarParser.java:1750) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation_expr(StellarParser.java:278)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation(StellarParser.java:147) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:89) ~[stormjar.jar:?]
> ... 19 more
> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
> org.apache.metron.common.dsl.ParseException: Unable to execute:
> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
> cannot be cast to java.util.List
> at org.apache.metron.common.stellar.StellarCompiler.
> getResult(StellarCompiler.java:428) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:90) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(
> DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(
> DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
> ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> doExecute(ProfileBuilderBolt.java:164) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> execute(ProfileBuilderBolt.java:144) [stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.
> outlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
> java.util.List
> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctio
> ns$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.StellarCompiler.
> exitTransformationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser$
> TransformationFuncContext.exitRule(StellarParser.java:1617)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_
> operands(StellarParser.java:1750) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation_expr(StellarParser.java:278)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation(StellarParser.java:147) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:89) ~[stormjar.jar:?]
> ... 19 more
>
>
>
> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <cestella@gmail.com> wrote:
>
>> Any chance there's a full stacktrace associated with that NPE anywhere?
>> Also, I assume this is 0.3.0, right?
>>
>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <shoggi@gmail.com> wrote:
>>
>>> Hi all
>>>
>>> Has anyone come across following profiler error, when trying to work
>>> with the MAD functions? The configuration loads fine and I only get the
>>> error when the profiler is pushing its data. I don't think its a typo but
>>> maybe the logic is not sound.
>>>
>>> Error from the profiler worker:
>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>> profile: profile=sketchy_mad, entity=krxd.net
>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>>> failure: message='null', tuple='source: __system:-1, stream: __tick, id:
>>> {}, [900]'
>>> java.lang.NullPointerException
>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>> java.lang.NullPointerException
>>>
>>>
>>> My Profiler config:
>>>
>>>     {
>>>       "profile": "sketchy_mad",
>>>       "foreach": "if exists(domain_without_subdomains) then
>>> domain_without_subdomains else 'n/a'",
>>>       "onlyif": "exists(domain_without_subdomains) && source.type ==
>>> 'squid'",
>>>       "init" : {
>>>         "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>       },
>>>       "tickUpdate" : {
>>>         "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>       },
>>>       "update": {
>>>         "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>       },
>>>       "result": "s"
>>>     }
>>>
>>> Regards
>>> shoggi
>>>
>>
>>
>

Mime
View raw message