commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Priblouda <>
Subject Re: [codec] Streamable Codec Framework
Date Mon, 03 Nov 2003 10:03:48 GMT

> Hi All,


> I noticed Alexander Hvostov's recent email
> containing streamable
> base64 codecs.  Given that the current codec
> implementations are
> oriented around in-memory buffers, is there room for
> an
> alternative codec framework supporting stream
> functionality?  I
> realise the need for streamable codecs may not be
> that great but
> it does seem like a gap in the current library.

I'm in the need. So we are at least 3 :) 

> Some of the goals I was working towards were:
> 1. No memory allocation during streaming.  This
> eliminates
> garbage collection during large conversions.
Cool. I got large conversions... I'm already at
mediumblob in mysql , and it goes up/down XML stream

> 2. Pipelineable codecs.  This allows multiple codecs
> to be chained
> together and treated as a single codec.  This allows
> codecs such as
> base 64 to be broken into two components (base64 and
> line wrapping
> codecs).

Also nice. 

> 2. Single OutputStream, InputStream implementations
> which
> utilise codec engines internally.  This eliminates
> the need to
> produce a buffer based engine and a stream engine
> for every codec.
> Note that this requires codec engines to be written
> in a manner
> that supports streaming.

If stream basedengine is there, it's not a problem to 
work on buffers... Though some codecs with internal
state may be tricky. 

> 3. Customisable receivers.  All codecs utilise
> receivers to
> handle conversion results.  This allows different
> outputs such as
> streams, in-memory buffers, etc to be supported.

And writers :) Velocity directives use them.

> 4. Direction agnostic codecs.  Decoupling the engine
> from the
> streams allows the engines to be used in different
> ways than
> originally intended.  Ie. You can perform base64
> encoding
> during reads from an InputStream.
> I have produced base64 and ascii hex codecs as a
> proof of concept
> and to evaluate performance.  It isn't as fast as
> the current
> buffer based codecs but is unlikely to ever be as
> fast due to the
> extra overheads associated with streaming.
> Both base64 and ascii hex implementations can
> produce a data rate
> of approximately 40MB/sec on a Pentium Mobile 1.5GHz
> notebook.
> With some performance tuning I'm sure this could be
> improved,
> I think array bounds checking is the largest
> performance hit.
> Currently requires jdk1.4 (exception handling
> requires rework
> for jdk1.3).
> Running ant without arguments in the root directory
> will build
> the project, run all unit tests and run performance
> tests.  Note
> that the tests require junit to be available within
> ant.
> Javadocs are the only documentation at the moment.
> Files can be found at:
> I hope someone finds this useful.  I'm not trying to
> force my
> implementation on anybody and I'm sure it could be
> improved in
> many ways.  I'm simply putting it forward as an
> optional approach.
> If it is decided that streamable codecs are a useful
> addition to
> commons I'd be glad to help.

I'll give it a look at and come back later today :)


----[ Konstantin Pribluda ( ko5tik ) ]----------------
Zu Verstärkung meines Teams suche ich ab Sofort einen
Softwareentwickler[In] für die Festanstellung. 
Arbeitsort: Mainz 
Skills:  Programieren, Kentnisse in OpenSource-Bereich
----[ ]------------------------

Do you Yahoo!?
Exclusive Video Premiere - Britney Spears

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message