eagle-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (EAGLE-1041) Support policy processing pipeline
Date Tue, 13 Jun 2017 12:27:00 GMT

    [ https://issues.apache.org/jira/browse/EAGLE-1041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047796#comment-16047796
] 

ASF GitHub Bot commented on EAGLE-1041:
---------------------------------------

GitHub user qingwen220 opened a pull request:

    https://github.com/apache/eagle/pull/947

    EAGLE-1041: Support policy processing pipeline

    https://issues.apache.org/jira/browse/EAGLE-1041
    
    Two updates:
    * if an inputStream is an intermediate stream (defined by select clause statement), then
remove it from inputStream list and outputStream list
    * if an inputStream is an intermediate stream, remove its PartitionSpec 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/qingwen220/eagle EAGLE-1041

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/eagle/pull/947.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #947
    
----
commit ea47b849700a89e15bc6a2e6b28317ee7a50b315
Author: Zhao, Qingwen <qingwzhao@apache.org>
Date:   2017-06-13T12:22:15Z

    update alertEditCtrl.js

----


> Support policy processing pipeline
> ----------------------------------
>
>                 Key: EAGLE-1041
>                 URL: https://issues.apache.org/jira/browse/EAGLE-1041
>             Project: Eagle
>          Issue Type: Improvement
>    Affects Versions: v0.5.0
>            Reporter: Zhao, Qingwen
>            Assignee: Zhao, Qingwen
>
> In some cases, like increment or decrement pattern, data need to be processed in more
than one stage. For example, alert if the metric value increases by N. Two steps to get the
right alert.
> 1. sort & filter events which meet the filter conditions.
> 2. define data change pattern. 
> Here is a sample policy
> {code}
> fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min)
select * group by site,host,component, metric insert into temp;
> from every a=HADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"],
> b=HADOOP_JMX_METRIC_STREAM_SANDBOX[b.component==a.component and b.metric==a.metric and
b.host==a.host and b.value>a.value and a.value>100]
> select b.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks, a.value
as oldNumOfMissingBlocks, (b.value-a.value) as increastedNumOfMissingBlocks insert into HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
> {code}
> There are two queries in this policy. The first one with the time window condition tells
Eagle to sort the original events. The second one defines the data pattern. As the constraint
of Siddhi syntax (https://docs.wso2.com/display/CEP420/SiddhiQL+Guide+3.1#SiddhiQLGuide3.1-Pattern),
the filtering of input events does not work. 
> Luckily, if we put the output stream of the first query as the input stream of the second
query, it works. That's the problem this ticket tries to solve. 
> Ideally, the right policy can be written as 
> {code}
> fromHADOOP_JMX_METRIC_STREAM_SANDBOX[metric=="hadoop.namenode.dfs.missingblocks"]#window.externalTime(timestamp,1min)
select * group by site,host,component, metric insert into MISSING_BLOCK_OUT;
> from every a=MISSING_BLOCK_OUT[metric=="hadoop.namenode.dfs.missingblocks"],
> b=MISSING_BLOCK_OUT[b.component==a.component and b.metric==a.metric and b.host==a.host
and b.value>a.value and a.value>100]
> select b.site,b.host,b.component, b.metric, b.value as newNumOfMissingBlocks, a.value
as oldNumOfMissingBlocks, (b.value-a.value) as increastedNumOfMissingBlocks insert into HADOOP_JMX_METRIC_STREAM_SANDBOX_MISS_BLOCKS_LARGER_OUT;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message