streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Beth Lavender <lavender.b...@gmail.com>
Subject Re: Question about processing architecture
Date Thu, 06 Jun 2013 12:08:53 GMT
On Tue, Jun 4, 2013 at 8:32 AM, Matt Franklin <m.ben.franklin@gmail.com>wrote:

> On Mon, Jun 3, 2013 at 10:52 PM, Jason Letourneau
> <jletourneau80@gmail.com>wrote:
>
> > The current vision is that filters will be implemented agnostic to the
> > overall processing architecture - there may be subscribers using lucene
> dsl
> > as part of the initial streams implemetation- but the interface won't
> > dictate how a subscriber filters it's activities -
>
>
> Maybe I am misunderstanding your statement.  In my mind, we really need
> inbound and outbound data pipelines.  I don't think a simple outbound
> filter can solve this easily.  I don't see how the system can do
> de-duplication, supersession, aggregation, etc during the outbound phase.
>  We will need to do a lot of processing before we hit an intermediate
> persistence layer that can then be used by subscriber filters and query
> endpoints.
>
> The pipeline components themselves should be pluggable and we just need a
> series of workflow events that they can hook and do work against the
> incoming data.
>
> Am I off base?
>

The rollup [1] reference is useful.  There is an implied set use cases
given the context for "views that support roll-up".  Do we have a set of
use cases documented (or that can be referenced) that would help drive
where in the architecture the plug ins are needed?

>
>
> > I don't know that we've
> > figured out whether the subscriber delegate tells the aggregate its
> filter
> > via an interface or whether aggregator tells each subscriber about every
> > activity and the subscriber filters - either way - you can implement
> > filters however you want - provides they adhere to the common filter
> > interface (which to my recollection is very simplistic)
> >
> > On Monday, June 3, 2013, Lavender, Beth A wrote:
> >
> > > Many of our current systems that will feed the integrated activity
> stream
> > > are noisy. For example, if I update a page 4 times in  5 minutes it
> > > generates an activity for each one.   I want to be able to set rule for
> > > discard the last n activities if they have the same actor, verb, and
> > object
> > >  in x time frame.
> > >
> > > This assumes a sub processor that detects the pattern and takes an
> action
> > > described in a rule. Where do rules and sub processors fit in this
> > > architecture? Is anyone doing this in their existing systems?
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message