commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brett Henderson" <jaka...@bretth.com>
Subject RE: [codec] StatefulDecoders
Date Wed, 10 Mar 2004 21:51:51 GMT
> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: Monday, 8 March 2004 3:25 PM
> To: Jakarta Commons Developers List
> Subject: RE: [codec] StatefulDecoders

> Consider the Cocoon (http://cocoon.apache.org/) pipeline for the 
> concept of
> 
> a fully event-driven programming approach, although their 
> implementation has
> 
> far too much overhead for codec purposes (or the regex events I 
> mentioned).

I still intend to look at this although my list of reading seems to grow
daily ...

> IMO, we want a consistent interface that provides the fundamental
> 
> operations, and then we can build convenience on top of that interace.
> 
> 
> 
> Your interface is closer to what I had in mind than Alex's, at least 
> using
> 
> more of the generic terminology.  The codec domain can be expressed 
> using
> 
> the pipeline interface, or if we want a codec specific interface, 
> Alex's
> 
> could be a convenience layer on top of the pipeline.
> 
> 
> 
> What I had imagined is an approach where each element in the pipeline
> 
> supports a registering for variety of event notifications. Some of 
> them may
> 
> be generic, some of them may be domain specific.  Most codec uses 
> would use
> 
> generic events.
> 
> 
> 
> The key is being able to go an object and register for semantic 
> events.  So
> 
> if you assume that a transformer is both a producer and a consumer, 
> you
> 
> could register a transformer with a datasource (producer), and 
> register
> 
> downstream consumers that want the decoded data with the transformer.  
> Yes,
> 
> we would want to allow both fan-in and fan-out where appropriate.

I think my design already supports most of the above ideas, they just have
to be implemented as required for the particular usage.  For example, a
Base64Encoder already supports registering for events of type byte[].  A
MIMEMultipartDecoder could generate events of type MIMEPart.  The event
types are not specified by the existing implementation, they can be added as
necessary for the particular feature.  Fan out is achieved by creating a
multicast stage accepting single events from a producer and passing them to
multiple consumers of the same type (although I haven't implemented a
multicast stage because I haven't needed it yet :-).  Fan in can already be
handled by setting a single consumer as the destination for multiple
producers.

Perhaps this isn't quite what you're envisaging.  You may like to see a more
generic approach that allows events and pipelines to be described in more
abstract terms.  Unfortunately I can't see a way of achieving this without
making the API complex and imposing overhead.  If you're looking for a more
powerful approach, should it be implemented outside of codec where runtime
issues aren't quite as critical?

I guess it depends on what problems you're trying to solve.  If you wish to
process large streams of data in an efficient manner my implementation is a
good fit, if you're looking to process structured data (eg. MIME) it can be
extended to fit as required, if you're looking to use it as the basis of
communication and processing within a server then it isn't up to the task.
However, isn't the last point outside the scope of codec and more in the
realm of other designs/libraries such as SEDA.

> As for the details of message transport ... it seems to me that we 
> already
> 
> have multiple options, so I'm not sure that we want to roll our own 
> versus
> 
> adopting and/or adapting an existing one.
> 
> 
> 
> We have JMS, and the new concurrency package coming in JDK 1.5.  One 
> thing
> 
> that got botched in JDK 1.5 is that they removed the Putable and 
> Takable
> 
> interfaces that Doug Lea had used in his library, instead merging 
> their
> 
> functionality directly onto the queue interface, falsely believing 
> that a
> 
> message queue is a Collection.  A number of us argued for those 
> interfaces,
> 
> and Doug proposed a change, but it was vetoed by Sun.
> 
> 
> 
> I see a few options, such as:
> 
> 
> 
>   - we pick up the necessary interfaces from Doug's concurrent
> 
>     library, and deal with java.util.concurrent down the road.
> 
> 
> 
>   - we use JMS interfaces in a very simplied form.
> 
> 
> 
> As potentially whacked as the idea might be, considering the 
> complexity of
> 
> JMS, I believe that we could selectively use JMS interfaces without 
> undo
> 
> complexity or hurting performance.  Basically, we'd ignore the things 
> that
> 
> don't make any sense in our context.  Take a look at MessageProducer,
> 
> MessageConsumer and MessageListener.  Intelligently, they are just
> 
> interfaces.  We don't need multi-threading, network transports, etc., 
> in
> 
> general, although by using those interfaces, they would be available 
> where
> 
> applications warranted them.
> 
> 
> 
> ref:
> 
> http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/packag
> e-summary.html
> 
> 
> 
> Alternatively, I had this odd thought that we could use the Callable
> 
> interface present in both Doug's code and java.util.concurrent.
> 
> 
> 
>   JSR-166:
> 
> http://gee.cs.oswego.edu/dl/concurrent/dist/docs/java/util/concurrent/
> package-summary.html



>> A structured content API can extend the producer/consumer ideas by 
>> passing

>> data types understood by the structured content in question.


While I don't have any hard and fast ideas, I believe codec would be best
suited with a custom set of interfaces for the following reasons: 1. They
can support all the features and only the feature that codec requires. 2.
They have no dependencies on any external libraries or JDK versions. 3. They
can copy the relevant terminology and ideas from the above libraries as
required. 4. Support for external libraries and transports can be added with
the relevant adapters keeping the core code insulated from external
dependencies.


> See also the hierarchical Topic support in JMS.



> As you know, we just had a JMS implementation offered, and there is

> somnifugi.  Do you think it is worth looking at an in-process JMS

> implementation for the pipeline?
I really think JMS is too complex for our purposes.  JMS input and output
adaptors would be very useful but using JMS as the method of connecting
stages is complex and adds dependencies to codec.  I couldn't see projects
such as HttpClient accepting a dependency on JMS libraries just so that they
can base64 encode some data.

However, I hadn't considered JMS prior to this point, Producers and
Consumers allowing data to be sent via JMS queues would be very useful if
queued or distributed processing was required.  Offloading data to another
machine for encoding then receiving the results asynchronously on another
queue would be very cool!

I apologise if I seem a little attached to my current implementation.

Cheers,
Brett


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org


Mime
View raw message