metron-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: Median Absolute Deviation - Profiler Error
Date Tue, 28 Feb 2017 02:34:29 GMT
Sorry, waht does the current profile config look like for that very last
error?

On Mon, Feb 27, 2017 at 6:39 PM, shoggi <shoggi@gmail.com> wrote:

> Seems not to be the culprit..
>
>
> 2017-02-28 00:35:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
> profile: profile=sketchy_mad, entity=optimicdn.com
> 2017-02-28 00:35:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
> failure: message='Unable to execute: org.apache.metron.statistics.outlier.
> MedianAbsoluteDeviationFunctions$State cannot be cast to java.util.List',
> tuple='source: __system:-1, stream: __tick, id: {}, [900]'
> org.apache.metron.common.dsl.ParseException: Unable to execute:
> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
> cannot be cast to java.util.List
> at org.apache.metron.common.stellar.StellarCompiler.
> getResult(StellarCompiler.java:428) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:90) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.execute(
> DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.assign(
> DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$null$2(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$flush$4(ProfileBuilderBolt.java:258)
> ~[stormjar.jar:?]
> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
> ~[?:1.8.0_77]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
> ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> doExecute(ProfileBuilderBolt.java:164) ~[stormjar.jar:?]
> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.
> execute(ProfileBuilderBolt.java:144) [stormjar.jar:?]
> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.
> outlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
> java.util.List
> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctio
> ns$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.StellarCompiler.
> exitTransformationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser$
> TransformationFuncContext.exitRule(StellarParser.java:1617)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1640)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.StellarParser.arithmetic_
> operands(StellarParser.java:1750) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr_mul(StellarParser.java:1537)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.arithmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation_expr(StellarParser.java:278)
> ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.generated.
> StellarParser.transformation(StellarParser.java:147) ~[stormjar.jar:?]
> at org.apache.metron.common.stellar.BaseStellarProcessor.
> parse(BaseStellarProcessor.java:89) ~[stormjar.jar:?]
> ... 19 more
>
> On Tue, Feb 28, 2017 at 12:18 AM, Casey Stella <cestella@gmail.com> wrote:
>
>> Ok, try something for me and tell me if it fixes it: adjust the config to
>> remove the tickUpdate (it didn't work anyway and we have since removed it
>> in 0.3.1).
>>
>>
>> On Mon, Feb 27, 2017 at 6:07 PM, shoggi <shoggi@gmail.com> wrote:
>>
>>> Yes, this is 0.3.0.
>>>
>>> Here is the full trace.
>>>
>>> 2017-02-28 00:05:43.367 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>> profile: profile=sketchy_mad, entity=krxd.net
>>> 2017-02-28 00:05:43.374 o.a.m.p.b.ProfileBuilderBolt [ERROR] Unexpected
>>> failure: message='Unable to execute: org.apache.metron.statistics.o
>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>> java.util.List', tuple='source: __system:-1, stream: __tick, id: {}, [900]'
>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>> cannot be cast to java.util.List
>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>> ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>> [stormjar.jar:?]
>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>> java.util.List
>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>> ~[stormjar.jar:?]
>>> ... 19 more
>>> 2017-02-28 00:05:43.377 o.a.s.d.executor [ERROR]
>>> org.apache.metron.common.dsl.ParseException: Unable to execute:
>>> org.apache.metron.statistics.outlier.MedianAbsoluteDeviationFunctions$State
>>> cannot be cast to java.util.List
>>> at org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:428)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:90)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.ex
>>> ecute(DefaultStellarExecutor.java:148) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.stellar.DefaultStellarExecutor.as
>>> sign(DefaultStellarExecutor.java:94) ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$nu
>>> ll$2(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.LinkedHashMap.forEach(LinkedHashMap.java:676) ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.lambda$fl
>>> ush$4(ProfileBuilderBolt.java:258) ~[stormjar.jar:?]
>>> at java.util.concurrent.ConcurrentMap.forEach(ConcurrentMap.java:114)
>>> ~[?:1.8.0_77]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.flush(ProfileBuilderBolt.java:237)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.doExecute(ProfileBuilderBolt.java:164)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.profiler.bolt.ProfileBuilderBolt.execute(ProfileBuilderBolt.java:144)
>>> [stormjar.jar:?]
>>> at org.apache.storm.daemon.executor$fn__6571$tuple_action_fn__6573.invoke(executor.clj:734)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$mk_task_receiver$fn__6492.invoke(executor.clj:469)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$clojure_handler$reify__6005.onEvent(disruptor.clj:40)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.daemon.executor$fn__6571$fn__6584$fn__6637.invoke(executor.clj:853)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484)
>>> [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
>>> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
>>> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
>>> Caused by: java.lang.ClassCastException: org.apache.metron.statistics.o
>>> utlier.MedianAbsoluteDeviationFunctions$State cannot be cast to
>>> java.util.List
>>> at org.apache.metron.statistics.outlier.MedianAbsoluteDeviation
>>> Functions$StateUpdate.apply(MedianAbsoluteDeviationFunctions.java:85)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.StellarCompiler.exitTransfo
>>> rmationFunc(StellarCompiler.java:246) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser$Tra
>>> nsformationFuncContext.exitRule(StellarParser.java:1617)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
>>> ~[stormjar.jar:?]
>>> at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
>>> ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.fun
>>> ctions(StellarParser.java:1640) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_operands(StellarParser.java:1750) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr_mul(StellarParser.java:1537) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.ari
>>> thmetic_expr(StellarParser.java:1397) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation_expr(StellarParser.java:278) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.generated.StellarParser.tra
>>> nsformation(StellarParser.java:147) ~[stormjar.jar:?]
>>> at org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:89)
>>> ~[stormjar.jar:?]
>>> ... 19 more
>>>
>>>
>>>
>>> On Mon, Feb 27, 2017 at 11:33 PM, Casey Stella <cestella@gmail.com>
>>> wrote:
>>>
>>>> Any chance there's a full stacktrace associated with that NPE
>>>> anywhere?  Also, I assume this is 0.3.0, right?
>>>>
>>>> On Mon, Feb 27, 2017 at 5:29 PM, shoggi <shoggi@gmail.com> wrote:
>>>>
>>>>> Hi all
>>>>>
>>>>> Has anyone come across following profiler error, when trying to work
>>>>> with the MAD functions? The configuration loads fine and I only get the
>>>>> error when the profiler is pushing its data. I don't think its a typo
but
>>>>> maybe the logic is not sound.
>>>>>
>>>>> Error from the profiler worker:
>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [INFO] Flushing
>>>>> profile: profile=sketchy_mad, entity=krxd.net
>>>>> 2017-02-27 23:20:43.368 o.a.m.p.b.ProfileBuilderBolt [ERROR]
>>>>> Unexpected failure: message='null', tuple='source: __system:-1, stream:
>>>>> __tick, id: {}, [900]'
>>>>> java.lang.NullPointerException
>>>>> 2017-02-27 23:20:43.368 o.a.s.d.executor [ERROR]
>>>>> java.lang.NullPointerException
>>>>>
>>>>>
>>>>> My Profiler config:
>>>>>
>>>>>     {
>>>>>       "profile": "sketchy_mad",
>>>>>       "foreach": "if exists(domain_without_subdomains) then
>>>>> domain_without_subdomains else 'n/a'",
>>>>>       "onlyif": "exists(domain_without_subdomains) && source.type
==
>>>>> 'squid'",
>>>>>       "init" : {
>>>>>         "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>> domain_without_subdomains, 5, 'MINUTES'))"
>>>>>       },
>>>>>       "tickUpdate" : {
>>>>>         "s": "OUTLIER_MAD_STATE_MERGE(PROFILE_GET('sketchy_mad',
>>>>> domain_without_subdomains, 5, 'MINUTES'), s)"
>>>>>       },
>>>>>       "update": {
>>>>>         "s": "OUTLIER_MAD_ADD(s, bytes)"
>>>>>       },
>>>>>       "result": "s"
>>>>>     }
>>>>>
>>>>> Regards
>>>>> shoggi
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message