Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 41359 invoked from network); 28 Mar 2007 19:47:02 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 28 Mar 2007 19:47:02 -0000 Received: (qmail 6487 invoked by uid 500); 28 Mar 2007 19:47:02 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 6454 invoked by uid 500); 28 Mar 2007 19:47:02 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 6443 invoked by uid 99); 28 Mar 2007 19:47:02 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Mar 2007 12:47:02 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [69.55.225.129] (HELO ehatchersolutions.com) (69.55.225.129) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Mar 2007 12:46:54 -0700 Received: by ehatchersolutions.com (Postfix, from userid 504) id 08FFF30EFC43; Wed, 28 Mar 2007 13:46:34 -0600 (MDT) X-Spam-Checker-Version: SpamAssassin 3.1.1 (2006-03-10) on javelina X-Spam-Level: Received: from [172.27.73.244] (unknown [128.143.255.102]) by ehatchersolutions.com (Postfix) with ESMTP id B07DD30EFC18 for ; Wed, 28 Mar 2007 13:46:24 -0600 (MDT) Mime-Version: 1.0 (Apple Message framework v752.3) In-Reply-To: <359a92830703281211y4f18a30do8b7194f95337e09d@mail.gmail.com> References: <68780f5d0703281056m1eb89919j4692a13b9a60753d@mail.gmail.com> <359a92830703281211y4f18a30do8b7194f95337e09d@mail.gmail.com> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Erik Hatcher Subject: Re: index file size threshold affecting search performance? Date: Wed, 28 Mar 2007 15:46:19 -0400 To: java-user@lucene.apache.org X-Mailer: Apple Mail (2.752.3) X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00 autolearn=ham version=3.1.1 I've just built a 9.3G index (admittedly tons of stored data in there, 3.3M documents) and performance is amazing (through Solr). Erik On Mar 28, 2007, at 3:11 PM, Erick Erickson wrote: > This surprises me, I'm currently working with a 4G index, and the > improvement from when it was an 8G index was only 10% or so. > And it's plenty speedy. > > Are you hitting hardware limitations and perhaps swapping like > crazy? In which case, unless you split things across several > machines, I doubt it would help to make two smaller indexes. > > In sum, I really suspect that you're NOT hitting a Lucene limitation, > but it's something else about your system.... > > Best > Erick > > On 3/28/07, Scott Oshima wrote: >> >> So I assumed a linear decay of performance as an index got bigger. >> >> For some reason when going from an index size of 1.89 to 1.95 gigs >> dramatically increased cpu across all of our servers. >> >> I was thinking of splitting the 1.95 index into 2 separate >> indexes and >> using a multisearcher on those parts? >> >> thanks. >> >> -scott >> --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org