streams-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Letourneau <>
Subject Re: Question about processing architecture
Date Tue, 04 Jun 2013 12:45:27 GMT
No - that's all correct - I was just addressing that the way filters
are put together won't be restricted by the streams processing other words how different implementations and use
cases filter or implement filtering doesn't matter in the context of
the processing architecture.  Your statement of pluggable pipeline
components is right on - hence the messaging architecture/EIP.

On Tue, Jun 4, 2013 at 8:32 AM, Matt Franklin <> wrote:
> On Mon, Jun 3, 2013 at 10:52 PM, Jason Letourneau
> <>wrote:
>> The current vision is that filters will be implemented agnostic to the
>> overall processing architecture - there may be subscribers using lucene dsl
>> as part of the initial streams implemetation- but the interface won't
>> dictate how a subscriber filters it's activities -
> Maybe I am misunderstanding your statement.  In my mind, we really need
> inbound and outbound data pipelines.  I don't think a simple outbound
> filter can solve this easily.  I don't see how the system can do
> de-duplication, supersession, aggregation, etc during the outbound phase.
>  We will need to do a lot of processing before we hit an intermediate
> persistence layer that can then be used by subscriber filters and query
> endpoints.
> The pipeline components themselves should be pluggable and we just need a
> series of workflow events that they can hook and do work against the
> incoming data.
> Am I off base?
>> I don't know that we've
>> figured out whether the subscriber delegate tells the aggregate its filter
>> via an interface or whether aggregator tells each subscriber about every
>> activity and the subscriber filters - either way - you can implement
>> filters however you want - provides they adhere to the common filter
>> interface (which to my recollection is very simplistic)
>> On Monday, June 3, 2013, Lavender, Beth A wrote:
>> > Many of our current systems that will feed the integrated activity stream
>> > are noisy. For example, if I update a page 4 times in  5 minutes it
>> > generates an activity for each one.   I want to be able to set rule for
>> > discard the last n activities if they have the same actor, verb, and
>> object
>> >  in x time frame.
>> >
>> > This assumes a sub processor that detects the pattern and takes an action
>> > described in a rule. Where do rules and sub processors fit in this
>> > architecture? Is anyone doing this in their existing systems?
>> >
>> >

View raw message