flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "kieran ." <kieran0...@hotmail.com>
Subject CEP issue
Date Tue, 29 Nov 2016 14:18:40 GMT
Hello,

I am currently building a multi-tenant monitoring application and exploring the effectiveness
of different Complex Event Processors (CEP) and whether or not this would be a potential solution
for what I want to achieve. I have created a small test application which utilises Flink and
its CEP but I have come across some issues when dealing with a large number of metrics to
monitor when using patterns/pattern streams. Flink seems to operate as expected with one,
or several patterns each consuming it's own PatternStream, but as soon as more are introduced
the memory usage of Flink seems to rise rather quickly and eventually throw an OutOfMemoryError.
My initial idea was to create one pattern/pattern stream for each metric that I need to monitor,
but there could be many thousands of these.

I create the PatternStream per Pattern like this to monitor a metric:


  Pattern<MetricData, ?> pattern = Pattern.<MetricData> begin( patternName ).subtype(
MetricData.class )

                .where(

                (evt -> evt.getValues().get( "max" ).longValue() > 50.0

                        && evt.account_id.equals( accountName )) );


        check.withPattern( pattern )

                .withTimePeriod( Integer.valueOf( 1 ) )

                .withCooldown( Integer.valueOf( 1 ) )

                .withName( checkName )

                .withAlertStatus( AlertStatus.OK )

                .setPatternStream(CEP.pattern(messageStream.keyBy("account_id"), pattern));

To trigger these patterns, I use


        PatternSelectFunction<MetricData, MetricWarning> psf = new PatternSelectFunction<MetricData,
MetricWarning>()

        {

            @Override

            public MetricWarning select( Map<String, MetricData> map ) throws Exception

            {

                return new MetricWarning(map.get(patternKey), name, accountId);

            }


        };


        try

        {

            check.getPatternStream().select(psf);

        }

        catch( Exception exception )

        {

            exception.printStackTrace();

        }


The pattern in the above example is tied to a specific stream which would result in one stream
per pattern and this seems to be an issue using this approach. If it would be possible to
run one pattern stream and switching out the patterns when needed, then perhaps this would
be a viable solution. Am I approaching this in the right way by creating a stream for each
pattern?

Would it be possible to create a set of Pattern processors that could be run against a single
PatternStream or is there anything you could suggest which would allow me to do this with
Flink?

Thanks,
- Kieran

<http://aka.ms/weboutlook>

Mime
View raw message