eagle-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JiJun Tang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (EAGLE-348) Alert engine base on spark streaming
Date Thu, 30 Mar 2017 05:07:41 GMT

    [ https://issues.apache.org/jira/browse/EAGLE-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948432#comment-15948432

JiJun Tang commented on EAGLE-348:

We will put the spark alert engine code into https://github.com/1haodian/eagle repo.

> Alert engine base on spark streaming
> ------------------------------------
>                 Key: EAGLE-348
>                 URL: https://issues.apache.org/jira/browse/EAGLE-348
>             Project: Eagle
>          Issue Type: New Feature
>    Affects Versions: v0.6.0
>            Reporter: JiJun Tang
>            Assignee: Hao Chen
>            Priority: Minor
>             Fix For: v0.6.0
> {noformat}
> New alert engine architecture:
> 1)Metaservice:   a) get meta(policies,topologies,streamdefinitions and so on) 
> 2)Coordinator:   a) pull policies and figure out if polices are changed by Metaservice
and notify config consumer of config changes by ConfigBusService (powered by zk node cache)
>                  b) placePolicy depend on policy usage on alertbolt
> 3)UnitTopology:  a) connect CorrelationSpout(read all topics data from kafka) StreamRouterBolt(find
which alertbolt to execute) AlertBolt(execute policy on stream) AlertPublisherBolt (publish
execute result to db,email and so on)
> Let the alert engine to work on spark streaming,I think :
> 1) #1 and #2 can be worked on both storm and spark platform, can use broadcast to distribute
this service
> 2) we should rewirte CorrelationSpout,StreamRouterBolt,AlertBolt,AlertPublisherBolt to
Dstream and connect them as DAG,then submit into spark platform.
> 3) spark can dynamic allocation executor without restart 
> 4) in the future spark streaming can support add or delete kafka topic without restart
> {noformat}

This message was sent by Atlassian JIRA

View raw message