commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Tanaka <>
Subject Re: [PIPELINE] Questions about pipeline
Date Wed, 05 Nov 2008 23:13:40 GMT
Thanks for confirming the examples. I figure that fleshing this out now 
should speed development at a future time.

Tim Dudgeon wrote:
> Ken Tanaka wrote:
>> Tim Dudgeon wrote:
>>> See comments below.
>>> Tim
>>> 2. each connection is explicity defined and could have extra 
>>> attributes added in future (e.g. a disable attribute to disable 
>>> execution of that part of the pipeline.
>>> 3. The concept of input can probably be generalised to include the 
>>> "feed", allowing multiple feeds to be used (as discussed earlier in 
>>> this thread). e.g. stage1 would also have an input that would be the 
>>> feed.
>> Do you envision a stage with two inputs (aPort and bPort) waiting 
>> until there are inputs on both before its stageDriver invokes the 
>> process method? If stage5 needs two inputs, and stage2 provides 3 
>> values and stage3 provides 2 values, there are just 2 complete pairs 
>> of values. The third value from stage2 could wait indefinitely for a 
>> matching input from stage3. Currently stages run until their queue is 
>> empty, but with multiple inputs that could be imbalanced, it might be 
>> better to set the quit condition to any one queue is empty and all 
>> upstream stages claim to be complete. Any non-empty queues on exit 
>> can trigger a warning.
> Yes, there are multiple scenarios here. e.g.
> 1. pairwise processing.
> a1 + b1
> a2 + b2
> a3 + b3
> ...
> 2. combinatorial processing
> a1 + b1
> a1 + b2
> a1 + b3
> a2 + b1
> a2 + b2
> ....
> (maybe others too)
> Clearly 1's inputs has to be matched, and if they are not some exit 
> rules would be required.
> Also, as you have mentioned before there are potential memory issues 
> with combinatorial processing of large sets of data, but I think 
> solutions can be found for this.
In general I prefer to stick to scenario 1. Any deviation can be handled 
by making a stage behave differently rather than putting this capability 
into the framework. A stage can be written to do nothing but accept 
pairwise arguments and emit combinatorial output, it can then be 
inserted before any stage to which you want to add combinatorial input 


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message