From java-user-return-53272-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org Tue Jul 17 21:07:34 2012 Return-Path: X-Original-To: apmail-lucene-java-user-archive@www.apache.org Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DEAD8D946 for ; Tue, 17 Jul 2012 21:07:34 +0000 (UTC) Received: (qmail 88313 invoked by uid 500); 17 Jul 2012 21:07:32 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 88272 invoked by uid 500); 17 Jul 2012 21:07:32 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 88263 invoked by uid 99); 17 Jul 2012 21:07:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jul 2012 21:07:32 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=FSL_RCVD_USER,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ssmith@mainstreamdata.com designates 209.63.42.202 as permitted sender) Received: from [209.63.42.202] (HELO nexus2.mainstreamdata.com) (209.63.42.202) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 17 Jul 2012 21:07:24 +0000 Received: from localhost (localhost [127.0.0.1]) by nexus2.mainstreamdata.com (Postfix) with ESMTP id A782A400D3 for ; Tue, 17 Jul 2012 15:07:02 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d= mainstreamdata.com; h=mime-version:content-transfer-encoding :content-type:content-type:content-language:accept-language :in-reply-to:references:message-id:date:date:subject:subject :from:from:received:received:received; s=key1; t=1342559221; bh= aGWHh8fIdUf8RvMChfoxtXH2MK2rHK0Mr8PkDgaIWow=; b=IEwXqo0i1x2XRRb0 DBT6PKluBsDQXYuMZnNgYC/5RU9podKdaK1VpDet4v8J2ExgmxKYgxakoxUz0nU8 nhLJ8S/79MSehVrGXIYJa8Xywv+Ycb0NEXYajVj5UxkBzscaru+KXjTLbhiqYcqs DOyr6AMlqN4Gyg9bIRpIWRGy1yFLEt4viWszHyQYuVhC2fTrIcFNBPer3mKpuknt 4FUZxOhaYi9V1Bb+BR7SNQmEw8aV8zv2hz2WfNmXWhWqIQf/MqShb9Fmy/HxPfXu PVmXD4Kb7LCeG/5p6WK5SYTBMBYNqOZp5rVgO0BbH2umViRXvcG/oUjoDzIvNu5f uSWOwQ== X-Virus-Scanned: Debian amavisd-new at nexus2.mainstreamdata.com Received: from nexus2.mainstreamdata.com ([127.0.0.1]) by localhost (nexus2.mainstreamdata.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id hzyIpPruck+n for ; Tue, 17 Jul 2012 15:07:01 -0600 (MDT) Received: from webmail.mainstreamdata.com (hedwig.slc.mainstreamdata.com [10.2.1.133]) by nexus2.mainstreamdata.com (Postfix) with ESMTPS id 9B469400D2 for ; Tue, 17 Jul 2012 15:07:01 -0600 (MDT) Received: from hedwig.slc.mainstreamdata.com ([10.2.1.133]) by hedwig.slc.mainstreamdata.com ([10.2.1.133]) with mapi id 14.01.0355.002; Tue, 17 Jul 2012 15:07:01 -0600 From: Scott Smith To: "java-user@lucene.apache.org" Subject: RE: Lucene reorganizing indexes Thread-Topic: Lucene reorganizing indexes Thread-Index: Ac1jjcktNyy6zw4bSkimaJwL1O1pTgAN/EyAACaWrDA= Date: Tue, 17 Jul 2012 21:07:01 +0000 Message-ID: References: <000c01cd6393$707f36a0$517da3e0$@gmx.de> In-Reply-To: <000c01cd6393$707f36a0$517da3e0$@gmx.de> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [217.24.9.235] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 It's lucene -----Original Message----- From: Ralf Heyde [mailto:ralf.heyde@gmx.de]=20 Sent: Monday, July 16, 2012 10:42 PM To: java-user@lucene.apache.org Subject: AW: Lucene reorganizing indexes=20 Do you use Lucene or Solr? We faced the problem in Solr due too big Caches, which where (re)warmed up = after a commit and the never ending full GCs. Greets Ralf -----Urspr=FCngliche Nachricht----- Von: Scott Smith [mailto:ssmith@mainstreamdata.com] Gesendet: Montag, 16. Juli 2012 22:29 An: java-user@lucene.apache.org Betreff: Lucene reorganizing indexes=20 We have an application that has to do "real time" indexing of a number of d= ocuments. What it does is wake up about every 20 seconds and updates the i= ndex with any changes that have been queued since the last time it ran. This involves adding and deleting several hundred documents. This is all d= one in a single thread. There can be multiple threads doing searches simul= taneous with the update thread (the searches run in a different process). Back in the days of 1.42, we would force an index optimization once each da= y. However, my impression is that the later versions of Lucene (we are cur= rently using 3.5), Lucene will often do its own reorganization based on hit= ting certain criteria. I've been told that optimizing the index is, perhap= s, no longer necessary. Can someone describe what happens here? The reason I'm asking about this is that we see our application periodicall= y using excessive amounts of kernel time (on Windows) which normally indica= tes a lot of disk activity. We are unable to align this with anything our = code is doing. Obviously, we expect Lucene to be causing disk activity, it= just seems that the last release (we were at 3.02 before going to 3.5) sev= erely increased the disk activity which is interfering with other things ru= nning on the boxes. Does any of this make sense to anyone? Is there an explanation? Thoughts = about what we might do about it? Thanks in advance. Scott --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org