commons-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Tanaka <Ken.Tan...@noaa.gov>
Subject Re: [PIPELINE] Questions about pipeline
Date Fri, 24 Oct 2008 18:34:24 GMT


Tim Dudgeon wrote:
> Ken Tanaka wrote:
>> The Pipeline Basics tutorial has now been incorporated into the 
>> project page. Thanks to some help and cleanup from Rahul Akolkar the 
>> documentation submitted was installed quickly. See
>>
>> http://commons.apache.org/sandbox/pipeline/pipeline_basics.html
>>
>> -Ken
>>
>
> That documentation is really useful. Thanks!
>
Wow, someone is actually looking at this. I'll work on cleaning up the 
documentation some. I hope people realize that some of the color-coded 
examples got some inadvertent newlines added--but this isn't relevant to 
your questions.
> Could I follow up one of the earlier questions in this thread on 
> branching and merging.
>
>
> From those docs it looks to me like the way data was set to a branch 
> is a bit strange. There appears to be a FileReaderStage class that has 
> Java bean property called htmlPipelineKey:
> <stage className="com.demo.pipeline.stages.FileReaderStage"
> driverFactoryId="df1" htmlPipelineKey="sales2html"/>
>
> and later in the pipeline a branch is defined that names the pipeline 
> according to that name:
> <pipeline key="sales2html">
>
> This seems pretty inflexible to me. Any branches have to be hardcoded 
> into the stage definition. I was expecting a situation where multiple 
> stages could be the recipients of the output of any stage, and these 
> can be "wired up" dynamically. e.g. something like this:
>
>
>          |--stage2
>          |
> stage1---+--stage3
>          |
>          |--stage4
>
> so that all you needed to do was to define a stage5 as one more 
> downstream stage for stage 1 and it would transparently receive the data.
>
> Is this possible, or does the branching have to be hard-coded into the 
> stage definition?
I wouldn't call the way branches are specified "hard coding", since the 
xml file here is a configuration file. For our current use, branches are 
pretty rare, so the pipeline framework deals best with simple cases that 
are fairly linear. Also, if stage1 is a branching stage, then that stage 
was written with branching in mind, and the "htmlPipelineKey" is a 
hard-coded property name in the stage source code, so it can direct 
output when it passes data out to the framework. To simplify matters, 
all your branching stages could follow a convention of using "branchKey" 
(or some other generic name), then you wouldn't have to remember what 
variable holds the branch name for which stage.

A stage could be written to take an arbitrary number of branch names, 
and thus send output down multiple branches, although it can get 
complicated configuring rules on what goes where if the same thing isn't 
going to all the branches. So rather than making stage1 a branching 
stage, it could be followed by "stageMulti", which would send copies of 
it's input to a number of outputs:

                  |-----stage2
                  |
stage1----stageMulti----stage3
                  |
                  |-----stage4

stageMulti could then be used to add branching to any stage it follows.

I can imagine making configuration files a little simpler with regards 
to setting up branching, but the more intelligent configuration file 
reader to handle that hasn't been written.
>
>
> Similarly for merging. To follow up the previous question, let say I 
> had stageA that output some A's and stage B that output some B's (lets 
> assume both A's and B's are simple numbers). Now I wanted to have a 
> stageC that takes all A's and all B's and generates some output with 
> the, (lets assume the output is A * B so that every combination of A * 
> B is output). So this would look like this:
>
> stageA--+
>         |
>         |----stageC
>         |
> stageB--+
>
> Is it possble to do this, so that stageA and stageB are both writing 
> to stageC, but that stageC can distinguish the 2 different streams of 
> data?
>
>
First off, the current design expects all pipelines to start with one 
stage, to accept feed values out of the config file (or place command 
line arguments into the first stage queue if the main pipeline 
application was been written to do that). So maybe you have a stageInit 
which takes a single number like "3"

feed "3" --> stageInit----stageA
                |
                ----------stageB

stageInit can then pass "3" on to stageA and stageB, possibly causing 
stageA to create 3 2-digit numbers and stageB to create 3 3-digit numbers.

For merging, stageC will accept normal input from a stage as well as 
watch for events carrying additional data. stageC may well have to 
accumulate input and then produce output as events are received. Stages 
normally accept one input, which is either a feed or the output of the 
stage immediately preceding them. Input from elsewhere or from more than 
one source is currently handled as events raised by the source and 
received by a "notify" method in the receiving stage.

feed "3" --> stageInit----stageA-------------stageC --> 10*111, 10*222, 
10*333, 20*111, 20*222, 20*333, 30*111....
                |      3          10, 20, 30    :
                ----------stageB................:
                       3          111, 222, 333
---- normal data flow
.... event passed data

Like branching, for our uses merging is rare. Also beware of running out 
of memory if you are doing any accumulation of data to merge input from 
more than one stage.

-Ken

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org


Mime
View raw message