directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ole Ersoy <>
Subject Re: Streaming / Serializing Big Objects
Date Fri, 08 Sep 2006 17:01:04 GMT

So if the decoded request request object is above the
configured threshold, then ADS would need to persist
it per the configured persitance mechanism(Prevayler,
...), otherwise we store it in memory.

The myfaces upload component looks at it's size
threshold and serializes the uploaded file if it's
above the specified threshold.  I'm sure it's just
uses Java serialization straight up, but the component
can be hooked up to any integration/persistance layer

Suppose the whole directly tree was stored using the
Eclipse EMF API.

The the decoder would map the request object directly
to a EMF object, and EMF's persistance mechanism could
be invoked to persist to xml, straight up object
serialization, the Service Data Object API could be
invoked to serialize to databases, etc.  Web Services
could be invoked, it's a pretty sexy API, with a lot
of possibilities.

When it comes to streaming images, resources, etc. I
would think the tomcat API's should be really good for

--- Emmanuel Lecharny <> wrote:

> Here is what we have to do to stream large objects :
> 1- Decoder :
> When we read the user request, we decode it from
> ASN.1 BER to a byte[] or to
> a String, depending of the object Type. But
> basically, we get a byte[].
> Whatever, we have two concerns :
>  A- if the length of this object - which is always
> known- is above a certain
> size (let say 1K), then we must store the object
> somwhere else than in
> memory. To do so, we must have a storage which can
> handle Strings, byte[]
> and StreamedObject[]. This has an impact on all 
> messages (we can't just
> work on some attributes, we have to be generic). So
> this is a huge
> refactoring, with accessors for those objects, and
> especially a
> accessor.
>  B- If we have to store a String (even a big one),
> we have to convert the
> byte[] to a String. If the String is big, then we
> must find a way to apply
> the byte[] -> String UTF8 conversion from a stream,
> and stream back the
> result. Not so easy ...
> 2- Database storage :
> Well, we now have decoded a request, and we have to
> store the value. The
> backend is not Stream ready at all. It should be
> able to handme a Stream and
> stores data without having to allocate a huge bunch
> of byte[].
> Another problem is the other operation : we read an
> entry from the backend,
> and we want a streamed data to remain streamed.
> Again, huge modification.
> 3- Encoder :
> Now, let suppose that we successfully get some data
> from the backend, and
> let's suppose that those data are streamed. We want
> to send them back to the
> client without having to create a big byte[]. That
> means we must be able to
> ask MINA to send chunks of data until we are done
> with the streamed data.
> ATM, what we do is that we write a full PDU - result
> of the encode() method
> - and MINA send it all. Here, the mechanism will be
> totally different : we
> should inform MINA to send some data as soon as we
> have a block of bytes
> ready (if we send 1500 bytes long blocks, then we
> may have to call MINA many
> times for a jpegPhoto.
> I may have forgotten some issues, so please tell me
> ! Regarding using a
> existing piece of code, I have to say : "well, why
> not ?". Right now, I
> think we should think seriously about the point I
> mentionned, and may be on
> a confluence page. Streaming will take at least 2
> weeks to write... Any
> already written piece of code that can help is ok :)
> Emmanuel
> On 9/8/06, Ole Ersoy <> wrote:
> >
> > I accidentally deleted the original message...
> >
> > The myfaces file upload component can be
> configured to
> > serialize objects larger than a specified size.
> >
> > If that sounds useful, I can extract some code...
> >
> > Cheers,
> > - Ole
> >
> > __________________________________________________
> > Do You Yahoo!?
> > Tired of spam?  Yahoo! Mail has the best spam
> protection around
> >
> >
> -- 
> Cordialement,
> Emmanuel L├ęcharny

Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 

View raw message