Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (asf.osuosl.org: local policy)
Message-ID: <43974BDA.9040203@apache.org>
Date: Wed, 07 Dec 2005 12:53:46 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla Thunderbird 1.0.7-1.1.fc3 (X11/20050929)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: Lucene performance bottlenecks
References: <3287125F0F0CCE4E8F55435208100CF534EE81@exchange-mbx.be.bvd>
	 <c68e39170512070728o64534585mbe6b01a713f58956@mail.gmail.com>
 <c68e39170512070743w3be062eal75ade9ce567f99d9@mail.gmail.com>
 <439742A9.9010000@getopt.org>
In-Reply-To: <439742A9.9010000@getopt.org>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Andrzej Bialecki wrote:
> It's nice to have these couple percent... however, it doesn't solve the 
> main problem; I need 50 or more percent increase... :-) and I suspect 
> this can be achieved only by some radical changes in the way Nutch uses 
> Lucene. It seems the default query structure is too complex to get a 
> decent performance.

That would certainly help.

For what it's worth, the Internet Archive has ~10M page Nutch indexes 
that perform adequately.  See:

http://websearch.archive.org/katrina/

The performance is about what you report, but it is quite usable. 
(Please don't stress-test this server!)  We recently built a ~100M page 
Nutch index at the Internet Archive that is surprisingly usable on a 
single CPU.  (This is not yet publicly accessible.)

Perhaps your traffic will be much higher than the Internet Archive's, or 
you have contractual obligations that specify certain average query 
performance, but, if not, ~10M pages is quite searchable using Nutch on 
a single CPU.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org