streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Franklin <>
Subject Re: Question about processing architecture
Date Tue, 04 Jun 2013 12:32:22 GMT
On Mon, Jun 3, 2013 at 10:52 PM, Jason Letourneau

> The current vision is that filters will be implemented agnostic to the
> overall processing architecture - there may be subscribers using lucene dsl
> as part of the initial streams implemetation- but the interface won't
> dictate how a subscriber filters it's activities -

Maybe I am misunderstanding your statement.  In my mind, we really need
inbound and outbound data pipelines.  I don't think a simple outbound
filter can solve this easily.  I don't see how the system can do
de-duplication, supersession, aggregation, etc during the outbound phase.
 We will need to do a lot of processing before we hit an intermediate
persistence layer that can then be used by subscriber filters and query

The pipeline components themselves should be pluggable and we just need a
series of workflow events that they can hook and do work against the
incoming data.

Am I off base?

> I don't know that we've
> figured out whether the subscriber delegate tells the aggregate its filter
> via an interface or whether aggregator tells each subscriber about every
> activity and the subscriber filters - either way - you can implement
> filters however you want - provides they adhere to the common filter
> interface (which to my recollection is very simplistic)
> On Monday, June 3, 2013, Lavender, Beth A wrote:
> > Many of our current systems that will feed the integrated activity stream
> > are noisy. For example, if I update a page 4 times in  5 minutes it
> > generates an activity for each one.   I want to be able to set rule for
> > discard the last n activities if they have the same actor, verb, and
> object
> >  in x time frame.
> >
> > This assumes a sub processor that detects the pattern and takes an action
> > described in a rule. Where do rules and sub processors fit in this
> > architecture? Is anyone doing this in their existing systems?
> >
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message