From java-user-return-45801-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org Wed Apr 14 21:28:46 2010 Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 24749 invoked from network); 14 Apr 2010 21:28:46 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 14 Apr 2010 21:28:46 -0000 Received: (qmail 1115 invoked by uid 500); 14 Apr 2010 21:28:44 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 1082 invoked by uid 500); 14 Apr 2010 21:28:44 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 1074 invoked by uid 99); 14 Apr 2010 21:28:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 21:28:44 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [85.25.71.29] (HELO mail.troja.net) (85.25.71.29) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Apr 2010 21:28:36 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.troja.net (Postfix) with ESMTP id 3D6C5D36006 for ; Wed, 14 Apr 2010 23:28:16 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mail.troja.net Received: from mail.troja.net ([127.0.0.1]) by localhost (megaira.troja.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id FfwnjNF1szH5 for ; Wed, 14 Apr 2010 23:28:11 +0200 (CEST) Received: from VEGA (port-83-236-62-54.dynamic.qsc.de [83.236.62.54]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.troja.net (Postfix) with ESMTPSA id C7765D36005 for ; Wed, 14 Apr 2010 23:28:10 +0200 (CEST) From: "Uwe Schindler" To: References: <1271268767.3543.64.camel@tpoljak-laptop> In-Reply-To: <1271268767.3543.64.camel@tpoljak-laptop> Subject: RE: NumericField indexing performance Date: Wed, 14 Apr 2010 23:28:17 +0200 Message-ID: <004b01cadc19$660a88e0$321f9aa0$@de> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acrb/i5ku5CAglPwRBubXUqdr66KRgAGdFMg Content-Language: de X-Virus-Checked: Checked by ClamAV on apache.org Hi Tomislav, indexing with NumericField takes longer (at least for the default = precision step of 4, which means out of 32 bit integers make 8 subterms = with each 4 bits of the value). So you produce 8 times more terms during = indexing that must be handled by the indexer. If you have lots of = documents, with distinct values the term index gets larger and larger, = but search performance increases dramatically (for NumericRangeQueries). = So if you index *only* numeric fields and nothing else, a 8 times slower = indexing can be true.=20 If you are not using NumericRangeQuery or you want tune indexing = performance, try larger precision Steps like 6 or 8. If you = don=E2=80=99t use NumericRangeQuery and only want to index the numeric = terms as *one* term, use precStep=3DInteger.MAX_VALUE. Also check your = memory requirements, as the indexer may need more memory and GC costs = too much. Also the index size will increase, so lots of more I/O is = done. Without more details I cannot say anything about your = configuration. So please tell us, how many documents, how many fields = and how many numeric fields in which configuration do you use? Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: uwe@thetaphi.de > -----Original Message----- > From: Tomislav Poljak [mailto:tpoljak@gmail.com] > Sent: Wednesday, April 14, 2010 8:13 PM > To: java-user@lucene.apache.org > Subject: NumericField indexing performance >=20 > Hi, > is it normal for indexing time to increase up to 10 times after > introducing NumericField instead of Field (for two fields)? >=20 > I've changed two date fields from String representation (Field) to > NumericField, now it is: >=20 > doc.add(new NumericField("time").setIntValue(date.getTime()/24/3600)) >=20 > and after this change indexing took 10x more time (before it was few > minutes and after more than an hour and half). I've tested with a > simple > counter like this: >=20 > doc.add(new NumericField("endTime").setIntValue(count++)) >=20 > but nothing changed, it still takes around 10x longer. If I comment > adding one numeric field to index time drops significantly and if I > comment both fields indexing takes only few minutes again. >=20 > Tomislav >=20 >=20 > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org > For additional commands, e-mail: java-user-help@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org