directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre-Arnaud Marcelot ...@marcelot.net>
Subject Re: Inex branch merged into trunk...
Date Wed, 02 May 2012 10:59:21 GMT
On 2 mai 2012, at 12:55, Kiran Ayyagari wrote:
> On Wed, May 2, 2012 at 4:20 PM, Pierre-Arnaud Marcelot <pa@marcelot.net> wrote:
>> On 2 mai 2012, at 12:24, Kiran Ayyagari wrote:
>>> this method has two extreme cases,
>>> 1. best performance when no attributes are requested
>>> 2. same overhead as cloning when all attributes are requested
>>> 
>>> cause in real world scenario 1 will never be used,
>> 
>> Actually in Studio, scenario 1 is used a lot.
>> That's exactly what happens do when opening a node and browsing the DIT in the UI
(almost, because we do ask for the objectClass attribute).
>> 
> yeap, I know, but this is a very limited usage, or shall I say just
> for tooling (essentially helpful for paging etc, else dOOM is certain
> the moment I expand a 100k node tree (provided my limit is set to >
> 100k ;))

Yeah, that's for sure.
We agree that directories are not meant to be browse by GUI as primary usage… ;)

Regards,
Pierre-Arnaud

>> Regards,
>> Pierre-Arnaud
>> 
>>> we might end up
>>> copying some attributes anyway
>>> but still better than cloning all
>>> On Wed, May 2, 2012 at 3:19 PM, Emmanuel Lécharny <elecharny@gmail.com>
wrote:
>>>> Le 5/2/12 9:53 AM, Alex Karasulu a écrit :
>>>>> 
>>>>> On Wed, May 2, 2012 at 2:43 AM, Emmanuel
>>>>> Lécharny<elecharny@gmail.com>wrote:
>>>>> 
>>>>>> Le 5/1/12 3:05 PM, Alex Karasulu a écrit :
>>>>>> 
>>>>>>> On Tue, May 1, 2012 at 4:08 AM, Emmanuel Lécharny<elecharny@gmail.com>**
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> o Object scope search (lookup) : 49 880 req/s compared to 23
081 on the
>>>>>>> previous trunk
>>>>>>> o One Level scope search (5 entries returned) : 68 715 entries
returned
>>>>>>> per second, compared to 33 120/s
>>>>>>> o Sub Level scope search (10 entries returned ) : 70 830 entries
>>>>>>> returned
>>>>>>> per second, compared to 18 910/s
>>>>>>> 
>>>>>>> 
>>>>>>> This is great work Emmanuel. Nicely done!
>>>>>>> 
>>>>>> I have some even better results, as of today :
>>>>>> 
>>>>>> o Object scope search (lookup) : 52 712 req/s compared to 23 081
on the
>>>>>> previous trunk
>>>>>> o One Level scope search (5 entries returned) : 72 635 entries returned
>>>>>> per second, compared to 33 120/s
>>>>>> o Sub Level scope search (10 entries returned ) : 75 100 entries
returned
>>>>>> per second, compared to 18 910/s
>>>>>> 
>>>>>> 
>>>>> This is just sick man! You've more than doubled the performance.
>>>> 
>>>> 
>>>> Some new idea this morning :
>>>> 
>>>> atm, we do clone the entries we fetch from the server, then we filter the
>>>> Attributes and the values, modifying the cloned entries. This leads to
>>>> useless create of the removed Attributes and Values. We suggested to
>>>> accumulate the modifications and to apply them at the end, avoiding the
>>>> cloning of AT which will not be returned.
>>>> 
>>>> First of all, we can avoid cloning the Values. The Value implementation are
>>>> immutable classes. This save around 7% of the time.
>>>> 
>>>> But this is not all we can do : we can simply avoid the accumulation of
>>>> modifications *and* avoid cloning the entry !
>>>> 
>>>> The idea is simple : when we get an entry in the cursor we have got back,
we
>>>> create a new empty entry, then we iterate over all the original entry's
>>>> attributes and values, and for each one of those attributes and values, we
>>>> check the filters, which will just tell if the Attribute/Value must be
>>>> ditched or kept. This way, we don't do anything useless, like storing the
>>>> modification or creating useless Attributes.
>>>> 
>>>> It will work to the extent we deal with the CollectiveAttributes which must
>>>> be injected somewhere, before we enter the loop (a connectiveAttribute might
>>>> perfectly be removed by the ACI filter...). But we can also inject those
>>>> added collective attributes into the loop of filters.
>>>> 
>>>> I may miss something, but I do think that this solution is a clear winner,
>>>> even in term of implementation...
>>>> 
>>>> thoughts ?
>>>> 
>>>>> 
>>>> 
>>>> 
>>>> --
>>>> Regards,
>>>> Cordialement,
>>>> Emmanuel Lécharny
>>>> www.iktek.com
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Kiran Ayyagari
>> 
> 
> 
> 
> -- 
> Kiran Ayyagari


Mime
View raw message