Mailing-List: contact dev-help@streams.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@streams.incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of lavender.beth@gmail.com
 designates 209.85.217.179 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAENpzYX33esc9j7jrjNZP9pKwEFmkwZZedFAaTAex9dMjJwwQA@mail.gmail.com>
References: <002C46140B2DCC4D8323F87F498E7A0426EF1B70@IMCMBX04.MITRE.ORG>
	<CAL8syOYTJSKOB_f0yUxCTNgYkeNbRUT5fLSE3kLCDn3XawCZ-Q@mail.gmail.com>
	<CAENpzYX33esc9j7jrjNZP9pKwEFmkwZZedFAaTAex9dMjJwwQA@mail.gmail.com>
Date: Thu, 6 Jun 2013 08:08:53 -0400
Message-ID: 
 <CAGif_KS5am1LXjCp1yx=pmxFd37b64LgqgC_yhghO0Jjg4QP4g@mail.gmail.com>
Subject: Re: Question about processing architecture
From: Beth Lavender <lavender.beth@gmail.com>
To: dev@streams.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11c3a9c88f461a04de7b2d45

--001a11c3a9c88f461a04de7b2d45
Content-Type: text/plain; charset=ISO-8859-1

On Tue, Jun 4, 2013 at 8:32 AM, Matt Franklin <m.ben.franklin@gmail.com>wrote:

> On Mon, Jun 3, 2013 at 10:52 PM, Jason Letourneau
> <jletourneau80@gmail.com>wrote:
>
> > The current vision is that filters will be implemented agnostic to the
> > overall processing architecture - there may be subscribers using lucene
> dsl
> > as part of the initial streams implemetation- but the interface won't
> > dictate how a subscriber filters it's activities -
>
>
> Maybe I am misunderstanding your statement.  In my mind, we really need
> inbound and outbound data pipelines.  I don't think a simple outbound
> filter can solve this easily.  I don't see how the system can do
> de-duplication, supersession, aggregation, etc during the outbound phase.
>  We will need to do a lot of processing before we hit an intermediate
> persistence layer that can then be used by subscriber filters and query
> endpoints.
>
> The pipeline components themselves should be pluggable and we just need a
> series of workflow events that they can hook and do work against the
> incoming data.
>
> Am I off base?
>

The rollup [1] reference is useful.  There is an implied set use cases
given the context for "views that support roll-up".  Do we have a set of
use cases documented (or that can be referenced) that would help drive
where in the architecture the plug ins are needed?

>
>
> > I don't know that we've
> > figured out whether the subscriber delegate tells the aggregate its
> filter
> > via an interface or whether aggregator tells each subscriber about every
> > activity and the subscriber filters - either way - you can implement
> > filters however you want - provides they adhere to the common filter
> > interface (which to my recollection is very simplistic)
> >
> > On Monday, June 3, 2013, Lavender, Beth A wrote:
> >
> > > Many of our current systems that will feed the integrated activity
> stream
> > > are noisy. For example, if I update a page 4 times in  5 minutes it
> > > generates an activity for each one.   I want to be able to set rule for
> > > discard the last n activities if they have the same actor, verb, and
> > object
> > >  in x time frame.
> > >
> > > This assumes a sub processor that detects the pattern and takes an
> action
> > > described in a rule. Where do rules and sub processors fit in this
> > > architecture? Is anyone doing this in their existing systems?
> > >
> > >
> >
>

--001a11c3a9c88f461a04de7b2d45--