Return-Path: X-Original-To: apmail-directory-dev-archive@www.apache.org Delivered-To: apmail-directory-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6712910DFE for ; Sun, 8 Sep 2013 03:58:46 +0000 (UTC) Received: (qmail 80789 invoked by uid 500); 8 Sep 2013 03:58:44 -0000 Delivered-To: apmail-directory-dev-archive@directory.apache.org Received: (qmail 80757 invoked by uid 500); 8 Sep 2013 03:58:31 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 80750 invoked by uid 99); 8 Sep 2013 03:58:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Sep 2013 03:58:26 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ayyagarikiran@gmail.com designates 74.125.82.181 as permitted sender) Received: from [74.125.82.181] (HELO mail-we0-f181.google.com) (74.125.82.181) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Sep 2013 03:58:22 +0000 Received: by mail-we0-f181.google.com with SMTP id p61so3101285wes.40 for ; Sat, 07 Sep 2013 20:58:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=zm7w0i1wBbq/alLzRE7dX2kewAL6Rg9KuIOeS05W/zc=; b=0jSjMuCqf4AC06QWVI0mO34Hn8xgdKTgIjp2JMW4Qbs6RLTpFRxWhRnPrAiLrv4VZ+ xNYe2bv1fFVLtxinN7eU7gv40qEskc6ewg7whmn2OQzemMoZdrTthBkJZ1uafrrqnYx0 jNo5esWkCZc2WrLDqZOSHPYGA2kdwS+Eg4L3BQrDt/DMHZgwCXbuzR/ij2Sj14bogGCn PVORjiTxvHZ8kPmrQV18JoWAp+Jm/AOkrErvLR6rxFnzyeYb0BBDOiCCI4shWsE/WFn8 gJat38XPrZX+XZDGcavI7l/y4mu+I6lEy0aRlhVEEOoj0+M6uzLUQUVT6VAo7c/e3D+A 55Aw== MIME-Version: 1.0 X-Received: by 10.180.182.68 with SMTP id ec4mr3875138wic.40.1378612680692; Sat, 07 Sep 2013 20:58:00 -0700 (PDT) Sender: ayyagarikiran@gmail.com Received: by 10.216.245.200 with HTTP; Sat, 7 Sep 2013 20:58:00 -0700 (PDT) In-Reply-To: <522ACDE7.3050602@gmail.com> References: <5227077D.6030207@gmail.com> <522ACDE7.3050602@gmail.com> Date: Sun, 8 Sep 2013 09:28:00 +0530 X-Google-Sender-Auth: 8iKTpzz-kaMUvwmL59X6ElcYtKc Message-ID: Subject: Re: Search performance potential improvements From: Kiran Ayyagari To: Apache Directory Developers List Content-Type: multipart/alternative; boundary=047d7b66f5991d9a7804e5d747b5 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b66f5991d9a7804e5d747b5 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On Sat, Sep 7, 2013 at 12:25 PM, Emmanuel L=E9charny w= rote: > Some more : in SubtreeScopeEvaluator.evaluate() and > OneLevelScopeEvaluator.evaluate(), we also check if the entry we will > work on is an alias : > > if ( null !=3D db.getAliasIndex().reverseLookup( id ) ) > { > return false; > } > > Replacing it with a cache check will again save 10%. > > FTR, by commenting this piece of code and the same in > DefaultSearchEngine, I reach 26 364 search/s on Mavibot (it was around > 19 000 before). > > that is a petty encouraging improvement, thanks for the heads up > Also note that since I started to work on performances, one month ago, > the gain is quite significant : up to 26 364 search/s from 5091/s... And > we still have some other ways to gain somemore performance :-) > > JDBM performances are 3 times behind (6 381/s) but it was 654/s on the > very first tests. > > > Le 9/4/13 12:12 PM, Emmanuel L=E9charny a =E9crit : > > Hi guys, > > > > I just came back from a short week of excellent vacations, and I spent > > the last two days looking at the search operation. > > > > Just before I left, I thought that the entry was cloned for no reason. > > The fact is that we do modify the entry before returning it, so we must > > modify a copy of the cached entry, otherwise we can have some really > > nasty bugs in other parts (the entry is cached, so we can't simply > > modify it). > > > > OTOH, I discovered that we do a costly check in the > > DefaultSearchEngine.computeResults() method : we check if the search > > BaseDN is an alias by looking into the Alias index. Not doing this chec= k > > imrpove the performance for around 15%. I'm pretty sure we can avoid > > looking into the index, as soon as we already have a cache of the > > existing aliases. The pb is that it's just a cache, and if the alias is > > not in the cache, we have to fetch the alias index. This become a pb > > when we have a lot of aliases, something unlikely to happen. > > > > I will try to think about a better solution. > > > > > > > -- > Regards, > Cordialement, > Emmanuel L=E9charny > www.iktek.com > > --=20 Kiran Ayyagari http://keydap.com --047d7b66f5991d9a7804e5d747b5 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable



On Sat, Sep 7, 2013 at 12:25 PM, Emmanuel L=E9charny <elecharny@= gmail.com> wrote:
Some more : in SubtreeScopeEvaluator.evaluat= e() and
OneLevelScopeEvaluator.evaluate(), we also check if the entry we will
work on is an alias :

=A0 =A0 =A0 =A0 if ( null !=3D db.getAliasIndex().reverseLookup( id ) )
=A0 =A0 =A0 =A0 {
=A0 =A0 =A0 =A0 =A0 =A0 return false;
=A0 =A0 =A0 =A0 }

Replacing it with a cache check will again save 10%.

FTR, by commenting this piece of code and the same in
DefaultSearchEngine, I reach 26 364 search/s on Mavibot (it was around
19 000 before).

that is a petty encouraging improvement, thanks for t= he heads up
Also note that since I started to work on performances, one month ago,
the gain is quite significant : up to 26 364 search/s from 5091/s... And we still have some other ways to gain somemore performance :-)

JDBM performances are 3 times behind (6 381/s) but it was 654/s on the
very first tests.


Le 9/4/13 12:12 PM, Emmanuel L=E9charny a =E9crit :
> Hi guys,
>
> I just came back from a short week of excellent vacations, and I spent=
> the last two days looking at the search operation.
>
> Just before I left, I thought that the entry was cloned for no reason.=
> The fact is that we do modify the entry before returning it, so we mus= t
> modify a copy of the cached entry, otherwise we can have some really > nasty bugs in other parts (the entry is cached, so we can't simply=
> modify it).
>
> OTOH, I discovered that we do a costly check in the
> DefaultSearchEngine.computeResults() method : we check if the search > BaseDN is an alias by looking into the Alias index. Not doing this che= ck
> imrpove the performance for around 15%. I'm pretty sure we can avo= id
> looking into the index, as soon as we already have a cache of the
> existing aliases. The pb is that it's just a cache, and if the ali= as is
> not in the cache, we have to fetch the alias index. This become a pb > when we have a lot of aliases, something unlikely to happen.
>
> I will try to think about a better solution.
>
>


--
Regards,
Cordialement,
Emmanuel L=E9charny
www.iktek.com




--
Kiran Ayyag= ari
http://keydap.com
--047d7b66f5991d9a7804e5d747b5--