directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Lecharny" <elecha...@gmail.com>
Subject Re: Streaming / Serializing Big Objects
Date Fri, 08 Sep 2006 13:27:40 GMT
Here is what we have to do to stream large objects :

1- Decoder :
When we read the user request, we decode it from ASN.1 BER to a byte[] or to
a String, depending of the object Type. But basically, we get a byte[].
Whatever, we have two concerns :
 A- if the length of this object - which is always known- is above a certain
size (let say 1K), then we must store the object somwhere else than in
memory. To do so, we must have a storage which can handle Strings, byte[]
and StreamedObject[]. This has an impact on all  messages (we can't just
work on some attributes, we have to be generic). So this is a huge
refactoring, with accessors for those objects, and especially a Stream.read()
accessor.
 B- If we have to store a String (even a big one), we have to convert the
byte[] to a String. If the String is big, then we must find a way to apply
the byte[] -> String UTF8 conversion from a stream, and stream back the
result. Not so easy ...

2- Database storage :
Well, we now have decoded a request, and we have to store the value. The
backend is not Stream ready at all. It should be able to handme a Stream and
stores data without having to allocate a huge bunch of byte[].
Another problem is the other operation : we read an entry from the backend,
and we want a streamed data to remain streamed. Again, huge modification.

3- Encoder :
Now, let suppose that we successfully get some data from the backend, and
let's suppose that those data are streamed. We want to send them back to the
client without having to create a big byte[]. That means we must be able to
ask MINA to send chunks of data until we are done with the streamed data.
ATM, what we do is that we write a full PDU - result of the encode() method
- and MINA send it all. Here, the mechanism will be totally different : we
should inform MINA to send some data as soon as we have a block of bytes
ready (if we send 1500 bytes long blocks, then we may have to call MINA many
times for a jpegPhoto.

I may have forgotten some issues, so please tell me ! Regarding using a
existing piece of code, I have to say : "well, why not ?". Right now, I
think we should think seriously about the point I mentionned, and may be on
a confluence page. Streaming will take at least 2 weeks to write... Any
already written piece of code that can help is ok :)

Emmanuel

On 9/8/06, Ole Ersoy <ole_ersoy@yahoo.com> wrote:
>
> I accidentally deleted the original message...
>
> The myfaces file upload component can be configured to
> serialize objects larger than a specified size.
>
> If that sounds useful, I can extract some code...
>
> Cheers,
> - Ole
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam?  Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>



-- 
Cordialement,
Emmanuel L├ęcharny

Mime
View raw message