Mailing-List: contact commons-dev-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Jakarta Commons Developers List" <commons-dev@jakarta.apache.org>
From: "Brett Henderson" <jakarta@bretth.com>
To: "'Jakarta Commons Developers List'" <commons-dev@jakarta.apache.org>
Subject: RE: [codec] StatefulDecoders
Date: Thu, 11 Mar 2004 08:51:51 +1100
Message-ID: <000601c406e9$e573e1a0$8263a88d@353661bh>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
In-Reply-To: <NBBBJGEAGJAKLIDBKJOPIEHPHNAB.noel@devtech.com>
Importance: Normal

> -----Original Message-----
> From: Noel J. Bergman [mailto:noel@devtech.com]
> Sent: Monday, 8 March 2004 3:25 PM
> To: Jakarta Commons Developers List
> Subject: RE: [codec] StatefulDecoders

> Consider the Cocoon (http://cocoon.apache.org/) pipeline for the=20
> concept of
>=20
> a fully event-driven programming approach, although their=20
> implementation has
>=20
> far too much overhead for codec purposes (or the regex events I=20
> mentioned).

I still intend to look at this although my list of reading seems to grow
daily ...

> IMO, we want a consistent interface that provides the fundamental
>=20
> operations, and then we can build convenience on top of that interace.
>=20
>=20
>=20
> Your interface is closer to what I had in mind than Alex's, at least=20
> using
>=20
> more of the generic terminology.  The codec domain can be expressed=20
> using
>=20
> the pipeline interface, or if we want a codec specific interface,=20
> Alex's
>=20
> could be a convenience layer on top of the pipeline.
>=20
>=20
>=20
> What I had imagined is an approach where each element in the pipeline
>=20
> supports a registering for variety of event notifications. Some of=20
> them may
>=20
> be generic, some of them may be domain specific.  Most codec uses=20
> would use
>=20
> generic events.
>=20
>=20
>=20
> The key is being able to go an object and register for semantic=20
> events.  So
>=20
> if you assume that a transformer is both a producer and a consumer,=20
> you
>=20
> could register a transformer with a datasource (producer), and=20
> register
>=20
> downstream consumers that want the decoded data with the transformer.  =

> Yes,
>=20
> we would want to allow both fan-in and fan-out where appropriate.

I think my design already supports most of the above ideas, they just =
have
to be implemented as required for the particular usage.  For example, a
Base64Encoder already supports registering for events of type byte[].  A
MIMEMultipartDecoder could generate events of type MIMEPart.  The event
types are not specified by the existing implementation, they can be =
added as
necessary for the particular feature.  Fan out is achieved by creating a
multicast stage accepting single events from a producer and passing them =
to
multiple consumers of the same type (although I haven't implemented a
multicast stage because I haven't needed it yet :-).  Fan in can already =
be
handled by setting a single consumer as the destination for multiple
producers.

Perhaps this isn't quite what you're envisaging.  You may like to see a =
more
generic approach that allows events and pipelines to be described in =
more
abstract terms.  Unfortunately I can't see a way of achieving this =
without
making the API complex and imposing overhead.  If you're looking for a =
more
powerful approach, should it be implemented outside of codec where =
runtime
issues aren't quite as critical?

I guess it depends on what problems you're trying to solve.  If you wish =
to
process large streams of data in an efficient manner my implementation =
is a
good fit, if you're looking to process structured data (eg. MIME) it can =
be
extended to fit as required, if you're looking to use it as the basis of
communication and processing within a server then it isn't up to the =
task.
However, isn't the last point outside the scope of codec and more in the
realm of other designs/libraries such as SEDA.

> As for the details of message transport ... it seems to me that we=20
> already
>=20
> have multiple options, so I'm not sure that we want to roll our own=20
> versus
>=20
> adopting and/or adapting an existing one.
>=20
>=20
>=20
> We have JMS, and the new concurrency package coming in JDK 1.5.  One=20
> thing
>=20
> that got botched in JDK 1.5 is that they removed the Putable and=20
> Takable
>=20
> interfaces that Doug Lea had used in his library, instead merging=20
> their
>=20
> functionality directly onto the queue interface, falsely believing=20
> that a
>=20
> message queue is a Collection.  A number of us argued for those=20
> interfaces,
>=20
> and Doug proposed a change, but it was vetoed by Sun.
>=20
>=20
>=20
> I see a few options, such as:
>=20
>=20
>=20
>   - we pick up the necessary interfaces from Doug's concurrent
>=20
>     library, and deal with java.util.concurrent down the road.
>=20
>=20
>=20
>   - we use JMS interfaces in a very simplied form.
>=20
>=20
>=20
> As potentially whacked as the idea might be, considering the=20
> complexity of
>=20
> JMS, I believe that we could selectively use JMS interfaces without=20
> undo
>=20
> complexity or hurting performance.  Basically, we'd ignore the things=20
> that
>=20
> don't make any sense in our context.  Take a look at MessageProducer,
>=20
> MessageConsumer and MessageListener.  Intelligently, they are just
>=20
> interfaces.  We don't need multi-threading, network transports, etc.,=20
> in
>=20
> general, although by using those interfaces, they would be available=20
> where
>=20
> applications warranted them.
>=20
>=20
>=20
> ref:
>=20
> http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/packag
> e-summary.html
>=20
>=20
>=20
> Alternatively, I had this odd thought that we could use the Callable
>=20
> interface present in both Doug's code and java.util.concurrent.
>=20
>=20
>=20
>   JSR-166:
>=20
> http://gee.cs.oswego.edu/dl/concurrent/dist/docs/java/util/concurrent/
> package-summary.html


>> A structured content API can extend the producer/consumer ideas by=20
>> passing

>> data types understood by the structured content in question.


While I don't have any hard and fast ideas, I believe codec would be =
best
suited with a custom set of interfaces for the following reasons: 1. =
They
can support all the features and only the feature that codec requires. =
2.
They have no dependencies on any external libraries or JDK versions. 3. =
They
can copy the relevant terminology and ideas from the above libraries as
required. 4. Support for external libraries and transports can be added =
with
the relevant adapters keeping the core code insulated from external
dependencies.


> See also the hierarchical Topic support in JMS.


> As you know, we just had a JMS implementation offered, and there is

> somnifugi.  Do you think it is worth looking at an in-process JMS

> implementation for the pipeline?
I really think JMS is too complex for our purposes.  JMS input and =
output
adaptors would be very useful but using JMS as the method of connecting
stages is complex and adds dependencies to codec.  I couldn't see =
projects
such as HttpClient accepting a dependency on JMS libraries just so that =
they
can base64 encode some data.

However, I hadn't considered JMS prior to this point, Producers and
Consumers allowing data to be sent via JMS queues would be very useful =
if
queued or distributed processing was required.  Offloading data to =
another
machine for encoding then receiving the results asynchronously on =
another
queue would be very cool!

I apologise if I seem a little attached to my current implementation.

Cheers,
Brett


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org