Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of dbichko@gmail.com designates
 72.14.220.152 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:content-transfer-encoding:content-disposition
         :references;
        b=gTLLXarz/kZu5jayKKJOT8ZuY5kiZ29Q+NPP/1rwo9t7VZZ19UhdoTm3MVaGSDKuuQ
         qTdycyfgjNAyFt14jB6fQKSxgi3KpviIWnVEkjo0FNQGWsgh1Nb1Yne3a3NDiFnDl9XI
         ULGfJOorx/H67GHUHpKQ8B33XbCQL/oYr7mig=
Message-ID: <74d027a50811111656i275cfe94j40e5d137eaea1fca@mail.gmail.com>
Date: Tue, 11 Nov 2008 19:56:10 -0500
From: "Dmitri Bichko" <dbichko@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: IndexSearcher and multi-threaded performance
In-Reply-To: <2DA1602E-72C8-454F-948B-CBEF408BB9E8@mikemccandless.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <74d027a50811111359y3b1355e4te2322c1e34e5e4d1@mail.gmail.com>
	 <2DA1602E-72C8-454F-948B-CBEF408BB9E8@mikemccandless.com>

I re-ran the no-readonly ram tests:

	thread	shared
1	64043	53610
2	26999	25260
3	27173	17265
4	22205	13222
5	20795	11098
6	17593	9852
7	17163	8987
8	17275	9052
9	19392	10266
10	27809	10397
11	25987	10724
12	26550	10832

The pattern is the same, but the difference at 4 and 6 is less
pronounced - it was probably just a hiccup (I'm not using terribly
sophisticated test methodology here), it's also possible I didn't give
the JVM enough RAM (this run was with 16GB, just to be on the safe
side).

Still, looks like the extra resource management overhead for
ram-thread beats whatever lock-contention ram-shared introduces.

I'm rerunning everything with readonly set and nio, I'll post the
results once it's done.

Cheers,
Dmitri

On Tue, Nov 11, 2008 at 5:40 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
>
> Nice results, thanks!
>
> The poor disk-based scaling may be fixed by NIOFSDirectory, if you are on
> Unix.  If you are on Windows it won't help (and will likely be worse than
> FSDirectory), because of an apparently bug in Sun's JVM on Windows whereby
> NIO positional file reads seem to share a lock under the hood.
>
> The poor ram-thread result  for 4 & 6 threads is odd.  Those numbers ought
> to be at least as good as ram-shared.  Is it possible those columns are
> swapped?  Because the ram-shared case should have been hurt by using a
> non-read-only IndexReader.
>
> Mike
>
> Dmitri Bichko wrote:
>
>> Hi,
>>
>> I'm pretty new to Lucene, so please bear with me if this has been
>> covered before.
>>
>> The wiki suggests sharing a single IndexSearcher between threads for
>> best performance
>> (http://wiki.apache.org/lucene-java/ImproveSearchingSpeed).  I've
>> tested running the same set of queries with: multiple threads sharing
>> the same searcher, with a separate searcher for each thread, both
>> shared/private with a RAMDirectory in-memory index, and (just for fun)
>> in multiple JVMs running concurrently (the results are in milliseconds
>> to complete the whole job):
>>
>> threads  multi-jvm  shared  per-thread  ram-shared  ram-thread
>>     1      72997   70883       72573       60308       60012
>>     2      33147   48762       35973       25498       25734
>>     4      16229   46828       21267       13127       27164
>>     6      13088   47240       14028        9858       29917
>>     8       9775   47020       10983        8948       10440
>>    10       8721   50132       11334        9587       11355
>>    12       7290   49002       11798        9832
>>    16       9365   47099       12338       11296
>>
>> The shared searcher indeed behaves better with a ram-based index, but
>> what's going on with the disk-based one?  It's basically not scaling
>> beyond two threads. Am I just doing something completely wrong here?
>>
>> The test consists of about 1,500 Boolean OR queries with 1-10
>> PhraseQueries each, with 1-20 Terms per PhraseQuery.  I'm using a
>> HitCollector to count the hits, so I'm not retrieving any results.
>> The index is about 5GB and 20 million documents.
>>
>> This is running on a 8 x quad-core Opteron machine with plenty of RAM to
>> spare.
>>
>> Any idea why I would see this behaviour?
>>
>> Thanks,
>> Dmitri
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org