Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:Received:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Up3s9tidyUUQ7BQS1T07xAD1GSJcZygGgYyUgtx7y1o4JhlCFqsj4XxHvHTNQo4YcA0yVRHPr/jv/lVWgibgrq3d7/H05ilLMNFZ8C8wr8oB5HLfBZzXVMI71GFKiYMQypxEPM3baXpTtStcbecU7HEmHRecFnQfWESxgfJzWhw=
  ;
Message-ID: <20060609000333.23411.qmail@web50305.mail.yahoo.com>
Date: Thu, 8 Jun 2006 17:03:33 -0700 (PDT)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Reply-To: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Re: IndexWriter.addIndexes & optimization
To: java-user@lucene.apache.org
In-Reply-To: <4487319D.10006@gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

When writing a unit test that comapres RAMDirectory and FSDirectory performance for Lucene in Action I had a very hard time showing that RAMDirectory really is faster. :)  Just set your maxBufferedDocs to as high a number as your RAM/heap will let you, and pick a mergeFactor that is high, but doesn't get you in trouble with open files.

Otis

----- Original Message ----
From: Dan Armbrust <daniel.armbrust.list@gmail.com>
To: java-user@lucene.apache.org
Sent: Wednesday, June 7, 2006 4:05:49 PM
Subject: Re: IndexWriter.addIndexes & optimization

Benjamin Stein wrote:

> 
> I could probably store the little RAMDirectories to disk as many
> FSDirectories, and then addIndexes() of *all* the FSDirectories at the end
> instead of every time.  That would probably be smart.
> 
> Glad I asked myself!
> 

That was what I was going to suggest - you may also want to benchmark to 
see if the RAMDirectory is buying you anything.  With the data that I am 
indexing on my hardware, I found it to be faster to index to a regular 
FSDirectory that it is to use the RAMDirectory.  Especially if you tweak 
the performance knobs on the indexer so it does its own caching before 
it writes to the Directory.

I do batches of documents to FSDirectories - and then merge all of the 
FSDirectories into a new master index at the end - so I never have to 
optimize during the indexing process.

Dan


-- 
****************************
Daniel Armbrust
Biomedical Informatics
Mayo Clinic Rochester
daniel.armbrust(at)mayo.edu
http://informatics.mayo.edu/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org