From java-user-return-49290-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org Mon Apr 04 20:00:22 2011 Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 53354 invoked from network); 4 Apr 2011 20:00:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 4 Apr 2011 20:00:22 -0000 Received: (qmail 69559 invoked by uid 500); 4 Apr 2011 20:00:18 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 69507 invoked by uid 500); 4 Apr 2011 20:00:18 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 69499 invoked by uid 99); 4 Apr 2011 20:00:18 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Apr 2011 20:00:18 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,FREEMAIL_REPLYTO_END_DIGIT,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of paul_t100@fastmail.fm designates 66.111.4.27 as permitted sender) Received: from [66.111.4.27] (HELO out3.smtp.messagingengine.com) (66.111.4.27) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Apr 2011 20:00:10 +0000 Received: from compute3.internal (compute3.nyi.mail.srv.osa [10.202.2.43]) by gateway1.messagingengine.com (Postfix) with ESMTP id 78F292063E; Mon, 4 Apr 2011 15:59:49 -0400 (EDT) Received: from frontend2.messagingengine.com ([10.202.2.161]) by compute3.internal (MEProxy); Mon, 04 Apr 2011 15:59:49 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d=messagingengine.com; h=message-id:date:from:reply-to:mime-version:to:cc:subject:references:in-reply-to:content-type:content-transfer-encoding; s=smtpout; bh=NWQnUb+2o5ATVohvugIewrQQj3c=; b=fBR08pp3HCOxDS9eeSGaVJ14B19PEZFPEkO88lHW3cNYaQ6sYxExOeV3oB85MOreLcUtzRNJF9C0beR66h7bGeiIBEyOiATfl4X5wvtKgtIuDuqiL0BFuWZSmAdyLzTAxgjs79VNx8mAsDSY1jjlMALgEtV6CPKqoYPRBwbWXWk= X-Sasl-enc: dT1Jbm8M3ib5cmWhwspRcasoq5NPJmSWm9BHHtx2TgGE 1301947188 Received: from [192.168.1.65] (unknown [217.155.98.246]) by mail.messagingengine.com (Postfix) with ESMTPSA id 7AF514426DB; Mon, 4 Apr 2011 15:59:48 -0400 (EDT) Message-ID: <4D9A2332.5070405@fastmail.fm> Date: Mon, 04 Apr 2011 20:59:46 +0100 From: Paul Taylor Reply-To: paul_t100@fastmail.fm User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Michael McCandless CC: java-user@lucene.apache.org Subject: Re: Lucene Merge failing on Open Files References: <4D9A168B.5020208@fastmail.fm> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org On 04/04/2011 20:13, Michael McCandless wrote: > How are you merging these indices? (IW.addIndexes?). > > Are you changing any of IW's defaults, eg mergeFactor? > > Mike Hi Mike I have indexWriter.setMaxBufferedDocs(10000); indexWriter.setMergeFactor(3000); these are a hangover from earlier code, I tried changing them and it didnt seem to make any difference, but do they look wrong ? The index that falls over during optmizationcreates about 10,000,000 records What I do is build 10 different indexes sequentially one after the after (so only one is hitting the db at any one time) but once the index is built I optimize the index in the background by creating an instance of the following class and submitting it to an ExecutionService , configured to have at most 2 threads acting I built my own class rather than just using optimize() with the background option because that wouldnt allow me to do the necessary debugging/calculations static class IndexWriterOptimizerAndClose implements Callable { private int maxId; private IndexWriter indexWriter; private DatabaseIndex index; private IndexOptions options; /** * * @param maxId * @param indexWriter * @param index * @param options */ public IndexWriterOptimizerAndClose(int maxId, IndexWriter indexWriter, DatabaseIndex index, IndexOptions options) { this.maxId=maxId; this.indexWriter= indexWriter; this.index=index; this.options=options; } public Boolean call() throws IOException, SQLException { StopWatch clock = new StopWatch(); clock.start(); String path = options.getIndexesDir() + index.getFilename(); System.out.println(index.getName()+":Started Optimization at "+Utils.formatCurrentTimeForOutput()); indexWriter.optimize(); indexWriter.close(); clock.stop(); // For debugging to check sql is not creating too few/many rows if(true) { int dbRows = index.getNoOfRows(maxId); IndexReader reader = IndexReader.open(FSDirectory.open(new File(path)),true); System.out.println(index.getName()+":"+dbRows+" db rows:"+(reader.maxDoc() - 1)+" lucene docs"); reader.close(); } System.out.println(index.getName()+":Finished Optimization:" + Utils.formatClock(clock)); return true; } } --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org