Return-Path: Delivered-To: apmail-jakarta-lucene-user-archive@www.apache.org Received: (qmail 56861 invoked from network); 22 Jul 2004 18:19:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur-2.apache.org with SMTP; 22 Jul 2004 18:19:52 -0000 Received: (qmail 32092 invoked by uid 500); 22 Jul 2004 18:19:45 -0000 Delivered-To: apmail-jakarta-lucene-user-archive@jakarta.apache.org Received: (qmail 32030 invoked by uid 500); 22 Jul 2004 18:19:44 -0000 Mailing-List: contact lucene-user-help@jakarta.apache.org; run by ezmlm Precedence: bulk List-Unsubscribe: List-Subscribe: List-Help: List-Post: List-Id: "Lucene Users List" Reply-To: "Lucene Users List" Delivered-To: mailing list lucene-user@jakarta.apache.org Received: (qmail 31997 invoked by uid 99); 22 Jul 2004 18:19:44 -0000 X-ASF-Spam-Status: No, hits=0.6 required=10.0 tests=NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [63.100.163.104] (HELO mercury.corp.cyveillance.com) (63.100.163.104) by apache.org (qpsmtpd/0.27.1) with ESMTP; Thu, 22 Jul 2004 11:19:41 -0700 Received: by mercury.corp.cyveillance.com with Internet Mail Service (5.5.2655.55) id ; Thu, 22 Jul 2004 14:19:08 -0400 Message-ID: <63434C14F9A6F74CB36B85033E4C30CA01393B@hermes.corp.cyveillance.com> From: wallen@Cyveillance.com To: lucene-user@jakarta.apache.org Subject: RE: Very slow IndexReader.open() performance Date: Thu, 22 Jul 2004 14:19:21 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2655.55) Content-Type: text/plain; charset="iso-8859-1" X-Virus-Checked: Checked X-Spam-Rating: minotaur-2.apache.org 1.6.2 0/1000/N It could also be that your disk space is filling up and the OS runs out of swap room. -----Original Message----- From: Mark Florence [mailto:mflorence@airsmail.com] Sent: Tuesday, July 20, 2004 1:52 PM To: Lucene Users List Subject: Very slow IndexReader.open() performance Hi -- We have a large index (~4m documents, ~14gb) that we haven't been able to optimize for some time, because the JVM throws OutOfMemory, after climbing to the maximum we can throw at it, 2gb. In fact, the OutOfMemory condition occurred most recently during a segment merge operation. maxMergeDocs was set to the default, and we seem to have gotten around this problem by setting it to some lower value, currently 100,000. The index is highly interactive so I took the hint from earlier posts to set it to this value. Good news! No more OutOfMemory conditions. Bad news: now, calling IndexReader.open() is taking 20+ seconds, and it is killing performance. I followed the design pattern in another earlier post from Doug. I take a batch of deletes, open an IndexReader, perform the deletes, then close it. Then I take a batch of adds, open an IndexWriter, perform the adds, then close it. Then I get a new IndexSearcher for searching. But because the index is so interactive, this sequence repeats itself all the time. My question is, is there a better way? Performance was fine when I could optimize. Can I hold onto singleton a IndexReader/IndexWriter/IndexSearcher to avoid the overhead of the open? Any help would be most gratefully received. Mark Florence, CTO, AIRS mflorence@airsmail.com 800-897-7714x1703 --------------------------------------------------------------------- To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org For additional commands, e-mail: lucene-user-help@jakarta.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org For additional commands, e-mail: lucene-user-help@jakarta.apache.org