directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject [jira] Commented: (DIR-30) submit StatefulDecoder to commons-codec
Date Sat, 21 Feb 2004 06:49:52 GMT
The following comment has been added to this issue:

     Author: Alex Karasulu
    Created: Fri, 20 Feb 2004 10:49 PM
The server is event driven and based on NIO non-blocking channels.  Hence incoming requests
may not arrive in one big complete chunk.  In fact reads from a client channel into a buffer
may contain any number of requests, and at most two partial requests.  Then again each chunck
may nicely contain a single complete PDU image.  You just never know how the client and the
server host operating systems are going to negotiate the transfers.  Also the size of the
message might be huge containing attributes representing massive blob's like high resolution
JPEG images taking the size of an entire screen or whatever.  Such a massive request would
take multiple packets, reads and buffers to make its way to the decoder.  The decoder must
correctly and efficiently handle these messages for the server to perform well under heavy

Multiple approaches can be taken for a decoder design.  Non-blocking IO makes localizing an
entire message PDU at one time difficult as seen above.  The decoder must be prepared to get
peices of the PDU at different times and hence manage a per request decoding session.  

The decode interface must provide a way to correlate chunk buffer contents with request decoding
sessions.  Here's how we propose to do that.

A decoder can take two approaches to decode a message.  The decoder can collect all the arriving
chunks and decode the entire PDU in one shot or decode the chunks as they arrive while maintaining
the decoding state for the request.  The first option is inefficient.  It's how virtually
every ASN.1 BER decoder works today and that's a shame.  The collect and decode in the one
shot method we'll refer to as the brut force method.  The brut force method wastes resources
and the waste is variable.  Consider the total footprint needed in memory to process a single
request of size N bytes.  The collection buffer would be N bytes, which represents the PDU's
transfer image.  When demarshalled into a message stub the image of the message probably takes
a tiny bit less memory than the actual transfer image.  Also the last chunk triggers the decode
and is held until the decoder finishes actively decoding.  We can reasonably conclude that
at the end of the decoding process a memory requirement of approximately 2N is needed for
each PDU of size N.  We can analyze this further within the appendix section.

The second approach where the decoder is a state machine started and stoped with the arrival
of more data.  This is what allows the streaming of content.  Servers designed with these
constructs are much more efficient and can hence support a higher degree of load.  Such a
decoder does not need the entire transfer image of the PDU in memory. However if it returns
the message it is still holding at least N + a chunk of memory when the decode completes.
 This is still variable and undesireable.

The problem is with having all the data within the message in memory.  This is tolerable for
small PDU's but not in the case of the PDU with massive JPEG attributes.  Holding the massive
transfer image in memory is wasteful.  The server's message framework can be designed to store
and lazy load large blobs of binary data to and from disk.  Furthermore these binary blobs
are not usually indexed nor are they used in filter expressions on a search: you wouldn't
use the bits of a friends JPEG to search for them in a directory would you?  So this info
can be streamed in and streamed out without having it all in memory at one time.

View this comment:

View the issue:

Here is an overview of the issue:
        Key: DIR-30
    Summary: submit StatefulDecoder to commons-codec
       Type: Task

     Status: In Progress
   Priority: Major

    Project: Directory

   Assignee: Alex Karasulu
   Reporter: Alex Karasulu

    Created: Fri, 20 Feb 2004 8:36 PM
    Updated: Fri, 20 Feb 2004 10:49 PM
    Due:     Fri, 20 Feb 2004 12:00 AM

Submit the StatefulDecoder interface to commons-codec.  Also ask if we can change DecoderException
so it extends IOException rather than a regular java.lang.Exception.  Most of these encode
and decode operations revolve around IO and it makes sense to see this as a form of IOException.
 In many respects a stateful decoder is really a filter which can be stacked.

This message is automatically generated by JIRA.

If you think it was sent incorrectly contact one of the administrators:

If you want more information on JIRA, or have a bug to report see:

View raw message