From dev-return-40847-apmail-directory-dev-archive=directory.apache.org@directory.apache.org Tue May 1 13:06:12 2012 Return-Path: X-Original-To: apmail-directory-dev-archive@www.apache.org Delivered-To: apmail-directory-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C43BB9B55 for ; Tue, 1 May 2012 13:06:12 +0000 (UTC) Received: (qmail 62874 invoked by uid 500); 1 May 2012 13:06:12 -0000 Delivered-To: apmail-directory-dev-archive@directory.apache.org Received: (qmail 62814 invoked by uid 500); 1 May 2012 13:06:12 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 62800 invoked by uid 99); 1 May 2012 13:06:12 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 May 2012 13:06:12 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of akarasulu@gmail.com designates 209.85.212.172 as permitted sender) Received: from [209.85.212.172] (HELO mail-wi0-f172.google.com) (209.85.212.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 May 2012 13:06:04 +0000 Received: by wibhj6 with SMTP id hj6so3711172wib.1 for ; Tue, 01 May 2012 06:05:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type; bh=iNrqqhUfll4JoEGCvUqUfntLrZVYM+IK1JWATGm9Usg=; b=hZWsgHvunb5xVxNtZ4duWSthSx1QAR1nFM9e3Yr5EBb3fR3xM55XInPi0iJUkDasw3 PCDsG4J0C0uBL2EBjFLPiY0kED0g7NIelevns+/mafxE03p9miwbohxzEpM9AnlfpHwF 0l5k+dA7HscLs8q4YyrZOpEUB4pgMOFvFGV0g1nogyP4RcgTK4jwBQZfVzX6Jl99gDR6 OJS3LSEC/Ug/9fzFkLL2nQQIdrFyo+JbjXR0yIfxmkxSvvCVuv8ku6xoBh4kgeJ39OdS 8yjJ9WI5Q8UkjNb2elK0XCwU4XWR1A42GQ4rjdrCrasDctABN4o9t1UVueIotUmZEpPh cQDQ== MIME-Version: 1.0 Received: by 10.180.100.230 with SMTP id fb6mr5910645wib.3.1335877543871; Tue, 01 May 2012 06:05:43 -0700 (PDT) Sender: akarasulu@gmail.com Received: by 10.180.126.97 with HTTP; Tue, 1 May 2012 06:05:43 -0700 (PDT) In-Reply-To: <4F9F3795.2040907@gmail.com> References: <4F9F3795.2040907@gmail.com> Date: Tue, 1 May 2012 16:05:43 +0300 X-Google-Sender-Auth: Xq8svhHUuHI1M9sI6cHpZptEAIs Message-ID: Subject: Re: Inex branch merged into trunk... From: Alex Karasulu To: Apache Directory Developers List , elecharny@apache.org Content-Type: multipart/alternative; boundary=f46d041824ee7745aa04bef93a63 --f46d041824ee7745aa04bef93a63 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Tue, May 1, 2012 at 4:08 AM, Emmanuel L=E9charny wr= ote: > Hi, > > just to inform you that the index branch has been merged with no harm > today. I just had to fix 3 conflicts, and two bugs I introduced in the > branch before the commit. > > The server performance is way better for searches, with a few improvement= s > I did those last 4 days. It was impressive how easy it was to improve the > speed with little modifications. The global result is that the server is > now : > o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the > previous trunk > o One Level scope search (5 entries returned) : 68 715 entries returned > per second, compared to 33 120/s > o Sub Level scope search (10 entries returned ) : 70 830 entries returned > per second, compared to 18 910/s > > This is great work Emmanuel. Nicely done! > There is room for more improvement, but it will be more complex. The area > that can be improved are : > o get rid of the extra getSearchControls() call in intercepotrs. This is > the easiest fix > o review the way we handle entries modification before we return them. > Currently, we clone the entry, and remove the attributes the user has not > required. See DIRSERVER-1719 for more explaination on this subject. Note > that the filtering of attributes represent around 9% of the global CPU ti= me. > o getting back the ID from a Dn is a very costly operation (19% of the > global CPU time), and the longer the DN, the longer the operation. For ea= ch > RDN, we have to do a lookup in the RdnIndex. The only solution would be t= o > have a Dn -> ID cache somewhere. This would boost the server performance, > that's for sure. > o fetching an entry from the backend cost 38% of the global time, out of > which 29% represent the cost to clone the entry. If we could avoid doing > this clone (see upper), we may have some major performances increase. > o when evaluating an entry to see if it fits the filter, we use the > reverseIndex, which is also a costly operation. We shoudl re-evaluate if = it > wouldn't be better to use the MatchingRules comparator to do that instead > (reverse lookups account for 4% of the used CPU time) > > I guess we have these in JIRA? > One interesting result is that the LRUCache.get() operation represent 13% > of the used time. This is definitively not small. There is probably some > room for some improvement here, but this is way more complex... > > All those numbers have been collected using YourKit on a Lookup test (150 > 000 lookups on one single element have been done) > > I wonder what the over the network stats are with a client machine separate from the server machine. Oh and with multiple clients. It's too bad we never got a chance to setup such an environment :( . > > There are also some improvements to expect on the Add/Delete/Move > operation, as we have to delete/add the keys on the RdnIndex. This is > something Im going to work on tomorrow. > > Cool. > > One more thing : the number I get when running the server-integ search > perf are way below (from 2900 to 5400 per second). This is plain normal. > When going through th network, we pay some extra price : > o the client code eats 57% of all the time it takes to run the test > o On the server, normalizing the incoming Dn costs 7% of the processing > time > o the entries encoding is very expensive > > All in all, on the server, unless we test it on a different machine than > the injectors, all the measures are pretty impossible to do. There is too > much noise... > > I'd be interested to conduct largest tests on a multi-core server, with > lots of memory, and a lot of entries, with external injectors, to see wha= t > kind of performances we can get... > > Ditto. > > In the next few days, I will probably fix some pending bugs. I think we > can cut a M7 release by the end of this week, and make it available by ne= xt > week. > > That sounds great. Thanks! --=20 Best Regards, -- Alex --f46d041824ee7745aa04bef93a63 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

On Tue, May 1, 2012 at 4:08 AM, Emmanuel= L=E9charny <elecharny@gmail.com> wrote:
Hi,

just to inform you that the index branch has been merged with no harm today= . I just had to fix 3 conflicts, and two bugs I introduced in the branch be= fore the commit.

The server performance is way better for searches, with a few improvements = I did those last 4 days. It was impressive how easy it was to improve the s= peed with little modifications. The global result is that the server is now= :
o Object scope search (lookup) : 49 880 req/s compared to 23 081 on the pre= vious trunk
o One Level scope search (5 entries returned) : 68 715 entries returned per= second, compared to 33 120/s
o Sub Level scope search (10 entries returned ) : 70 830 entries returned p= er second, compared to 18 910/s


This is great work Emmanuel. Nicely do= ne!
=A0
There is room for more improvement, but it will be more complex. The area t= hat can be improved are :
o get rid of the extra getSearchControls() call in intercepotrs. This is th= e easiest fix
o review the way we handle entries modification before we return them. Curr= ently, we clone the entry, and remove the attributes the user has not requi= red. See DIRSERVER-1719 for more explaination on this subject. Note that th= e filtering of attributes represent around 9% of the global CPU time.
o getting back the ID from a Dn is a very costly operation (19% of the glob= al CPU time), and the longer the DN, the longer the operation. For each RDN= , we have to do a lookup in the RdnIndex. The only solution would be to hav= e a Dn -> ID cache somewhere. This would boost the server performance, t= hat's for sure.
o fetching an entry from the backend cost 38% of the global time, out of wh= ich 29% represent the cost to clone the entry. If we could avoid doing this= clone (see upper), we may have some major performances increase.
o when evaluating an entry to see if it fits the filter, we use the reverse= Index, which is also a costly operation. We shoudl re-evaluate if it wouldn= 't be better to use the MatchingRules comparator to do that instead (re= verse lookups account for 4% of the used CPU time)


I guess we have these in JIRA?
=A0
One interesting result is that the LRUCache.get() operation represent 13% o= f the used time. This is definitively not small. There is probably some roo= m for some improvement here, but this is way more complex...

All those numbers have been collected using YourKit on a Lookup test (150 0= 00 lookups on one single element have been done)


I wonder what the over the network sta= ts are with a client machine separate from the server machine. Oh and with = multiple clients. It's too bad we never got a chance to setup such an e= nvironment :( .
=A0

There are also some improvements to expect on the Add/Delete/Move operation= , as we have to delete/add the keys on the RdnIndex. This is something Im g= oing to work on tomorrow.


Cool.
=A0

One more thing : the number I get when running the server-integ search perf= are way below (from 2900 to 5400 per second). This is plain normal. When g= oing through th network, we pay some extra price :
o the client code eats 57% of all the time it takes to run the test
o On the server, normalizing the incoming Dn costs 7% of the processing tim= e
o the entries encoding is very expensive

All in all, on the server, unless we test it on a different machine than th= e injectors, all the measures are pretty impossible to do. There is too muc= h noise...

I'd be interested to conduct largest tests on a multi-core server, with= lots of memory, and a lot of entries, with external injectors, to see what= kind of performances we can get...


Ditto.
=A0

In the next few days, I will probably fix some pending bugs. I think we can= cut a M7 release by the end of this week, and make it available by next we= ek.


That sounds great. Thanks!
=

--
Best Regards,
-- Alex

--f46d041824ee7745aa04bef93a63--