commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kris Nuttycombe" <>
Subject [chain] Pipeline implementation
Date Fri, 17 Sep 2004 17:28:07 GMT
Hi, all,

I'm writing to get some advice and perhaps offer some code that may be 
useful to the commons-chain project or elsewhere.

The group I work for does a large amount of data processing and we are 
working on solutions for pipelined data processing. Our current 
implementation uses a pipeline model where each stage in the pipeline 
has an integrated blocking queue and an abstract process(Object o) 
method that is sequentially applied to each element in the queue. When a 
stage is finished processing, it may pass the processed object (or 
products derived from it) onto the input queue of one or more subsequent 
stages in the pipeline. Branching pipelines are supported, and the whole 
mess is configured using Digester.

There's a lot of similarity here with the chain of responsibility 
pattern that commons-chain implements, but subtle differences as well. 
Each stage runs in one or more separate threads and we are working to 
allow the processing to be distributed across the network. The pipeline 
model assumes that each object placed in the pipe is going to be 
processed by every stage, whereas to my understanding the chain of 
responsibility is more designed for finding an appropriate command to 
use to process a given context. Also, the pipeline is designed to run as 
a service where data can be provided for processing by automated 
systems. For example, data being beamed down from a satellite can be 
aggregated into orbits that are then passed into the pipeline for 
generation of geolocated gridded products, statistical analysis, etc.

Our group would really like to be able to contribute some of this code 
back to the commons effort, since we use a ton of commons components. 
The amount of overlap with commons-chain is significant, but I'm not 
sure it's a perfect match because of the differing goals. Does anyone 
out there know of other similar efforts? Is there a place for this sort 
of code in commons? Are we just missing something fundamental about 
commons-chain where we should simply be using that instead?

Suggestions would be much appreciated. I'm happy to send code, examples, 
and documentation to anyone who's interested.


Kris Nuttycombe
Associate Scientist
Enterprise Data Systems Group
CIRES, National Geophysical Data Center/NOAA
(303) 497-6337

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message