Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of
 SRS0=RV_V5g=27=basetechnology.com=jack@yourhostingaccount.com designates
 65.254.254.77 as permitted sender)
Message-ID: <14714CF93FA04417B7B936078D373869@JackKrupansky14>
From: "Jack Krupansky" <jack@basetechnology.com>
To: <java-user@lucene.apache.org>
References: <01AFE0FB733B9944974A82A09CEB7A0309C81ABB21@mail3.imedx.com>
	<1400570841.2420.155.camel@te-prime>
	<01AFE0FB733B9944974A82A09CEB7A0309C835D126@mail3.imedx.com>
	<1400578244.2420.170.camel@te-prime>
	<01AFE0FB733B9944974A82A09CEB7A0309C835D16A@mail3.imedx.com>
	<1400581087.2420.182.camel@te-prime>
	<01AFE0FB733B9944974A82A09CEB7A0309C835D36D@mail3.imedx.com>
	<1401153243448-4138215.post@n3.nabble.com>
	<003301cf7974$7a200600$6e601200$@gmx.de>
	<01AFE0FB733B9944974A82A09CEB7A0309C881E6CF@mail3.imedx.com>
	<CAN4YXvcH=oKFmgrw2ZiGkiwOc2cq4RR1FQsq4E-Cw-nToA2Sqw@mail.gmail.com>
 <CAN4YXvf8oXwev4U8TErS3mmURwNuja6w2_PB=sxNYoKkqL2_4w@mail.gmail.com>
 <538C1F0F.3010300@mailarchiva.com>
In-Reply-To: <538C1F0F.3010300@mailarchiva.com>
Subject: Re: search performance
Date: Mon, 2 Jun 2014 07:34:49 -0400
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="UTF-8";
	reply-type=response
Content-Transfer-Encoding: 7bit
Importance: Normal
Sender: "Jack Krupansky" <jack@basetechnology.com>

Do you have enough system memory to fit the entire index in OS system memory 
so that the OS can fully cache it instead of thrashing with I/O? Do you see 
a lot of I/O or are the queries compute-bound?

You said you have a 128GB machine, so that sounds small for your index. Have 
you tried a 256GB machine?

How frequent are your commits for updates while doing queries?

-- Jack Krupansky

-----Original Message----- 
From: Jamie
Sent: Monday, June 2, 2014 2:51 AM
To: java-user@lucene.apache.org
Subject: search performance

Greetings

Despite following all the recommended optimizations (as described at
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed) , in some of
our installations, search performance has reached the point where is it
unacceptably slow. For instance, in one environment, the total index
size is 200GB, with 150 million documents indexed. With NRT enabled,
search speed is roughly 5 minutes on average. The server resources are:
2x6 Core Intel CPU, 128GB, 2 SSD for index and RAID 0, with Linux.

The only thing we haven't yet done, is to upgrade Lucene from 4.7.x to
4.8.x. Is this likely to make any noticeable difference in performance?

Clearly, longer term, we need to move to a distributed search model. We
thought to take advantage of the distributed search features offered in
Solr, however, our solution is very tightly integrated into Lucene
directly (since Solr didn't exist when we started out). Moving to Solr
now seems like a daunting prospect. We've also following the Katta
project with interest, but it doesn't appear support distributed
indexing, and development on it seems to have stalled. It would be nice
if there were a distributed search project on the Lucene level that we
could use.

I realize this is a rather vague question, but are there any further
suggestions on ways to improve search performance? We need cheap and
dirty ideas, as well as longer term advice on a possible path forward.

Much appreciate

Jamie

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org