directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lecharny <elecha...@gmail.com>
Subject Re: LDAP codec streaming
Date Fri, 28 Jul 2006 06:21:58 GMT
Norbet Reilly a écrit :

> Hi everyone,
>
> Thanks for my inclusion to the project as a "mentored commiter" (I
> didn't mind "proby" myself ;-), it's an honour to be involved with
> such an exciting opensource project.
>
> I was just wanting to ask one of the ADS gurus what happens exactly at
> the LDAP protocol level when an extremely large result set is returned
> by ADS for an LDAP search(). Is a client able to start retreiving
> results via their NamingEnumeration before the server has completed
> its search? 

Yep. We don't build the full result first, and then send it to the 
client. In fact, an iterator is created,
and while iterating the result, some entries are pushed back to the 
client as soon as they are fetched.

> I'm hoping there is some sort of "chunking" at the
> protocol level so that batches of results are streamed to the client
> as they become available, rather then the waiting for the entire
> result set to be available first. Is this the case?

It is :) (Alex can confirm)

But there is something that can be done to improve this mechanism : Big 
objects like jpeg image, for instance, are loaded in memory and sent 
back to the client. This is dangerous for the server, because if you 
have more than one client requesting for more than one image, then you 
can exhaust the memory. Especially if you consider caching, in which 
case entries are in memory, including their attributes.

Even worst, they will use twice the memory needed to store this image : 
one as an Attribute, and one as a byte[] in the SearchResultEntry 
encoded PDU that will be sent back to the client. This is also the case 
when the cleint send this object when adding it or modifying it.
We have thought of using a streaming mechanism allowing us to avoid to 
keep such objects in memory, with special accessors to get chunks of 
byte[] from a file instead of loading the full object in memory, but we 
never had time to build it.

Obviously, this is something we need. It goes deep into the server, but 
not that much. From Ldap point of view, we just need to modify the 
LockableAttributeImpl object to deal with a huge attribute, and in this 
case, stream it to a temporary location on the disk (let say that if an 
attribute length is longer than 1024 bytes - or char if it's a String -, 
then a temp file will be used. The trikiest part will be MINA. we are 
building a ByteBuffer to store the result to send, which will contain 
the full block of data. What can be done is to modify the way we 
cooperate with MINA to send chunks of data - 4096 bytes long? - and 
continue until all the baytes have been sent.

Here is a good piece of interesting work to be done :)

If you are interested, well, we also are to have it into the server! So 
just don't be afraid, this is not that complicated. We can provide 
assistance and guidance if needed.

Emmanuel

>
> Thanks
>


Mime
View raw message