From dev-return-33351-apmail-directory-dev-archive=directory.apache.org@directory.apache.org Sun May 09 16:46:57 2010 Return-Path: Delivered-To: apmail-directory-dev-archive@www.apache.org Received: (qmail 24218 invoked from network); 9 May 2010 16:46:57 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 9 May 2010 16:46:57 -0000 Received: (qmail 1438 invoked by uid 500); 9 May 2010 16:46:56 -0000 Delivered-To: apmail-directory-dev-archive@directory.apache.org Received: (qmail 1383 invoked by uid 500); 9 May 2010 16:46:56 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 1376 invoked by uid 99); 9 May 2010 16:46:56 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 16:46:56 +0000 X-ASF-Spam-Status: No, hits=0.9 required=10.0 tests=AWL,FREEMAIL_FROM,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of elecharny@gmail.com designates 74.125.82.50 as permitted sender) Received: from [74.125.82.50] (HELO mail-ww0-f50.google.com) (74.125.82.50) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 09 May 2010 16:46:51 +0000 Received: by wwb18 with SMTP id 18so958519wwb.37 for ; Sun, 09 May 2010 09:46:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:reply-to :user-agent:mime-version:to:subject:content-type :content-transfer-encoding; bh=7IXhhosGlsp6qpfLlkM38tHCjQtJNRAApI/Whv3/rpg=; b=WYyh/nJwV4NkhAhBqR1DplkhSGjsLwEN2J9ZBK2a5xtlWnYvyVQvlwGMHwb9yimIxC vngPp6x8GzseWkIwJlrXipMkiSr1Mu1StNFjDqhRqODF2FQDro1ZOo2S/5TvumnFpTN5 f/ASzS8whmQxvepbl0KFXEa9mIn7KTiB3vTSg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:reply-to:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; b=HrypL6Cy8UQ5cotaE403sBhlB9sxnCpF1CmG80s/BekwCESHuDKe0ZeySS2C+1qH2A 417FEl2cB9sOaDCtXWvldmE/A3JupYiSt1Jll42GVaTOw8MMBkEbf+OOjfx+CnUIEmcR +FcmeUHc4A1h2YFh9jVLTzAwcUH/TXBHbLfHo= Received: by 10.227.152.69 with SMTP id f5mr2674508wbw.73.1273423590714; Sun, 09 May 2010 09:46:30 -0700 (PDT) Received: from emmanuel-lecharnys-MacBook-Pro.local (ver78-3-89-80-150-189.dsl.club-internet.fr [89.80.150.189]) by mx.google.com with ESMTPS id u8sm33908089wbc.11.2010.05.09.09.46.29 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 09 May 2010 09:46:30 -0700 (PDT) Message-ID: <4BE6E6E5.4040903@gmail.com> Date: Sun, 09 May 2010 18:46:29 +0200 From: Emmanuel Lecharny Reply-To: elecharny@apache.org User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.1.9) Gecko/20100317 Thunderbird/3.0.4 MIME-Version: 1.0 To: Apache Directory Developers List Subject: Re: Add perf issues Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit >> Welcome to my nightmare. ;) > > I'm still looking for a fast/cheap index data structure that can be > sparsely populated, is lossless, and scales to billions of entries. > (Currently our index is fast/cheap but lossy. If you fill a slot with a > million entries and then remove every other one, it still records the slot > as containing a million entries. I.e. we really don't handle sparse indices > at those scales.) > What surprises me is that we were supposed to have solve this issue, by using external subtrees (we were able to load 25 millions of entries in a flat tree months ago). I guess we have done some modification lately that broke this... Regarding the other aspects of the problem, well, until we are able to inject thousands of entries again, I must say that having a loseless index is my last concern :) > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- Regards, Cordialement, Emmanuel Lécharny www.iktek.com