Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jamie@mailarchiva.com
 designates 82.145.44.153 as permitted sender)
Message-ID: <53A3E7FE.40603@mailarchiva.com>
Date: Fri, 20 Jun 2014 09:51:26 +0200
From: Jamie <jamie@mailarchiva.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9;
 rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: search performance
References: <01AFE0FB733B9944974A82A09CEB7A0309C81ABB21@mail3.imedx.com>
	<538C1F0F.3010300@mailarchiva.com>
	<CA+iSzr21AX7ZgdMHxGS7=nQqry9SupSqLjN8AFe5CZ1VJR8EfQ@mail.gmail.com>
	<538C5174.7040101@mailarchiva.com>
	<CALr4Hzr+qWTi-o2yyBp9Fz_m66oFYXEG=w9G2cT0ecxPUdv3_A@mail.gmail.com>
	<538D962E.2020809@mailarchiva.com>
	<CAPz8bx1PeKuvqgP=VqEnc2u8G9L=R_5TTJcQ5wj+VgfwBayFDw@mail.gmail.com>
	<538D99A4.1060605@mailarchiva.com>	<538D9BAC.7030501@mailarchiva.com>
	<CALr4HzpZe9AuKR6bjv4gik-m133DNV=3ONYkDUyi38V-9JXWzw@mail.gmail.com>
	<538DA1A3.4040302@mailarchiva.com>
	<CALr4Hzo-hhCxGOY8si_J9Vrf54nKHe_qOZp=_BhoGewcbMi28g@mail.gmail.com>
	<538DA6AE.4030402@mailarchiva.com>
	<CAOdYfZUxqJy1yVbywsLMwS65XNaWH-xNOkyELumpXdMBahCxDA@mail.gmail.com>
	<538DB0C4.5050701@mailarchiva.com>
	<CAOdYfZU6hmnt2Lhfy6OJbaz8yJwN0r1hf_Nxiw60ZESNxVdNWA@mail.gmail.com>
	<538DC3ED.4030407@mailarchiva.com>
 <CAOEyTYSOHduovCCccynwEWF_uuGgxQk+qBQ3t=QUteMG3YsG9A@mail.gmail.com>
 <53923B1C.6090200@mailarchiva.com>
In-Reply-To: <53923B1C.6090200@mailarchiva.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi All

Thank you for all your suggestions. Some of the recommendations hadn't 
yet been implemented, as our code base was using older versions of 
Lucene with reduced capabilities. Thus, far, all the recommendations  
for fast search have been implemented (e.g. using pagination with 
searchAfter, DirectoryReader.openIfChanged, avoiding wrapping lucene 
scoreDoc results, option to disable sorting, etc.).

While, in some environments, search performance has improved 
significantly, in other larger ones we are unfortunately, still seeing 1 
minute - 5 minute search times. For instance, in one site, the total 
index size is 500GB with 190 million documents indexed. They are running 
a machine with 24 core and 4 SSD drives to house the indexes. New emails 
are being added to the indexes at a rate of 10 message/sec.

One area possible area for improvement: Searching is being conducted 
across several indexes. To accomplish this, on each search, a 
MultiReader is constructed, that consists of several subreaders created 
by the DirectoryReader.openIfChangedMethod. Only one of the indexes is 
updated frequently, the others are never updated.  For each search, a 
new IndexSearcher is created passed the MultiReader in the constructor. 
 From what I've read, MultiReader and IndexSearcher are relatively 
lightweight and should not impact search performance. Is this correct? 
Is there a faster way to handle searching across multiple indexes? What 
is the performance impact of searching across multiple indexes?

Am I correct that using SearchManager can't be used with a MultiReader 
and NRT?  I would appreciate all suggestions on how to optimize our 
search performance further. Search time has become a usability issue.

Much appreciate

Jamie

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org