commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kris Nuttycombe" <>
Subject Re: [chain] Pipeline implementation
Date Fri, 17 Sep 2004 20:23:25 GMT
Hi, Alex,

There is definitely some overlap here, and I think that there's good 
potential for collaboration. Our current implementation of the pipeline 
can definitely benefit from the per-stage thread pool configuration that 
you've done, since right now each one of our stages is still 
single-threaded. Adding pooling to increase concurrency within the 
stages is something that's been on our drawing board for a while.

The model that we've been using is that stages process data instead of 
events, although one could certainly consider an event as a type of 
data.  Our stages have to be aware of the  pipelines in which they 
reside because our Stage interface defines additional "exqueue(Object 
obj)" and "exqueue(String key, Object obj)" methods which are used to 
enqueue an object on either a subsequent stage or a keyed pipeline 
branch, respectively.

Can you give me a little more information about how you're handling 
routing between the stages, or where to look in the source code?


Alex Karasulu wrote:

>This pipeline stage you describe sounds very much like a SEDA stage;
>seda stage = queue + thread pool + processing code.   Events are queued
>from one stage to another and you can change the way routing is
>handled.  The main difference is we're using these similar patterns to
>build a common framework for highly concurrent inet servers.  Sounds
>like what you're doing is more geared to generic processing piplelines.
>Kris and Craig is there any overlap here where we can possibly help each
>One of the biggest problems I have had to date is handling the
>serialization of certain events.  Is this something you have a nice
>solution for? 
>BTW Craig you spoke about getting together a SEDA package of sorts
>extracted from the directory source.  We're actually in the process of
>doing that.  It's a separate project all together.  It has not been
>integrated into the site yet but the code is present here:
>Specifically you might want to take a look at the stage package here:
>To unsubscribe, e-mail:
>For additional commands, e-mail:

Kris Nuttycombe
Associate Scientist
Geospatial Data Services Group
CIRES, National Geophysical Data Center/NOAA
(303) 497-6337

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message