Return-Path: 
 <java-user-return-47836-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-java-user-archive@www.apache.org
Received: (qmail 94351 invoked from network); 17 Nov 2010 21:57:58 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 17 Nov 2010 21:57:58 -0000
Received: (qmail 3668 invoked by uid 500); 17 Nov 2010 21:58:27 -0000
Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org
Received: (qmail 3634 invoked by uid 500); 17 Nov 2010 21:58:27 -0000
Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:java-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:java-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:java-user@lucene.apache.org>
List-Id: <java-user.lucene.apache.org>
Reply-To: java-user@lucene.apache.org
Delivered-To: mailing list java-user@lucene.apache.org
Received: (qmail 3626 invoked by uid 99); 17 Nov 2010 21:58:27 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 21:58:27 +0000
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=RCVD_IN_DNSWL_NONE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of
 mark.kristensson@smartsheet.com designates 209.85.212.176 as permitted
 sender)
Received: from [209.85.212.176] (HELO mail-px0-f176.google.com)
 (209.85.212.176)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 17 Nov 2010 21:58:22 +0000
Received: by pxi11 with SMTP id 11so484253pxi.35
        for <java-user@lucene.apache.org>;
 Wed, 17 Nov 2010 13:58:02 -0800 (PST)
Received: by 10.142.115.6 with SMTP id n6mr9389909wfc.169.1290031082265;
        Wed, 17 Nov 2010 13:58:02 -0800 (PST)
Received: from [10.0.2.104] (75-145-24-62-Washington.hfc.comcastbusiness.net
 [75.145.24.62])
        by mx.google.com with ESMTPS id x18sm3281599wfa.11.2010.11.17.13.57.59
        (version=TLSv1/SSLv3 cipher=RC4-MD5);
        Wed, 17 Nov 2010 13:58:00 -0800 (PST)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1081)
Subject: Re: IndexWriter.close() performance issue
From: Mark Kristensson <mark.kristensson@smartsheet.com>
In-Reply-To: <AANLkTikQx9OgpZw7RWZUdjNJrTtriKGO5Zv7F7EpNqxp@mail.gmail.com>
Date: Wed, 17 Nov 2010 13:57:54 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <778F13F3-C842-4132-AC5D-9FF31D721442@smartsheet.com>
References: <B0BF640B-3986-42D6-B081-9C8B51B8E07E@smartsheet.com>
 <AANLkTi=JbCXFPSNucwnY5OK4fKTfQojk_5yt0==avpr8@mail.gmail.com>
 <79A8B638-16FD-486C-BE62-76CCAE58D620@smartsheet.com>
 <8ED783FE-BBE2-415A-A3B0-847D7500509E@smartsheet.com>
 <AANLkTi=N-4VbdG1qDRUH+0rMyOjSQ8f_GTZJMJ_=_s4B@mail.gmail.com>
 <1D4E4B59-1D3E-4CC2-B655-2729861E179B@smartsheet.com>
 <AANLkTimJjZC8ZJYsKpSn50ER-nSUhmOG_ZMQYYwLVHA5@mail.gmail.com>
 <ACBD00C9-7962-4E60-9F5D-0C8A9A6A730C@smartsheet.com>
 <AANLkTinZcwETVu3VXLqhzBpin6UpP2c5e417V3nduaPs@mail.gmail.com>
 <1F455CF6-C0F6-48E0-906D-989B374BB23E@smartsheet.com>
 <AANLkTinyXmXnN3xftEugQnF3EAT+LmHeEe_64ytvojhy@mail.gmail.com>
 <AANLkTikQx9OgpZw7RWZUdjNJrTtriKGO5Zv7F7EpNqxp@mail.gmail.com>
To: java-user@lucene.apache.org
X-Mailer: Apple Mail (2.1081)

Sure,

There is only one stack trace (that seems to be how the output for this =
tool works) for java.lang.String.intern:

TRACE 300165:
	java.lang.String.intern(<Unknown Source>:Unknown line)
	=
org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.ja=
va:74)
	org.apache.lucene.util.StringHelper.intern(StringHelper.java:34)
	org.apache.lucene.index.FieldInfos.read(FieldInfos.java:353)
	org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
	=
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.jav=
a:116)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:576)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:554)
	=
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
	=
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRe=
ader.java:27)
	=
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:75)
	=
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.jav=
a:677)
	=
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:188)
	com.smartsheet.Main.main(<Unknown Source>:Unknown line)

But there are several for =
org.apache.lucene.util.SimpleStringInterner.intern and =
org.apache.lucene.util.StringHelper.intern:


TRACE 300252:
	=
org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.ja=
va:61)
	org.apache.lucene.util.StringHelper.intern(StringHelper.java:34)
	org.apache.lucene.index.FieldInfos.read(FieldInfos.java:353)
	org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
	=
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.jav=
a:116)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:576)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:554)
	=
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
	=
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRe=
ader.java:27)
	=
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:75)
	=
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.jav=
a:677)
	=
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:188)
	com.smartsheet.Main.main(<Unknown Source>:Unknown line)
TRACE 300339:
	=
org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.ja=
va:75)
	org.apache.lucene.util.StringHelper.intern(StringHelper.java:34)
	org.apache.lucene.index.FieldInfos.read(FieldInfos.java:353)
	org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
	=
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.jav=
a:116)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:576)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:554)
	=
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
	=
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRe=
ader.java:27)
	=
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:75)
	=
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.jav=
a:677)
	=
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:188)
	com.smartsheet.Main.main(<Unknown Source>:Unknown line)
TRACE 300262:
	org.apache.lucene.util.StringHelper.intern(StringHelper.java:34)
	=
org.apache.lucene.index.FieldInfos.addInternal(FieldInfos.java:249)
	org.apache.lucene.index.FieldInfos.read(FieldInfos.java:363)
	org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
	=
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.jav=
a:116)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:576)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:554)
	=
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
	=
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRe=
ader.java:27)
	=
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:75)
	=
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.jav=
a:677)
	=
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:188)
	com.smartsheet.Main.main(<Unknown Source>:Unknown line)
TRACE 300344:
	=
org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.ja=
va:78)
	org.apache.lucene.util.StringHelper.intern(StringHelper.java:34)
	org.apache.lucene.index.FieldInfos.read(FieldInfos.java:353)
	org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
	=
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.jav=
a:116)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:576)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:554)
	=
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
	=
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRe=
ader.java:27)
	=
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:75)
	=
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.jav=
a:677)
	=
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:188)
	com.smartsheet.Main.main(<Unknown Source>:Unknown line)
TRACE 300206:
	=
org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.ja=
va:54)
	org.apache.lucene.util.StringHelper.intern(StringHelper.java:34)
	org.apache.lucene.index.FieldInfos.read(FieldInfos.java:353)
	org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
	=
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.jav=
a:116)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:576)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:554)
	=
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
	=
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRe=
ader.java:27)
	=
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:75)
	=
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.jav=
a:677)
	=
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:188)
	com.smartsheet.Main.main(<Unknown Source>:Unknown line)
TRACE 300211:
	=
org.apache.lucene.util.SimpleStringInterner.intern(SimpleStringInterner.ja=
va:51)
	org.apache.lucene.util.StringHelper.intern(StringHelper.java:34)
	org.apache.lucene.index.FieldInfos.read(FieldInfos.java:353)
	org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)
	=
org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.jav=
a:116)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:576)
	=
org.apache.lucene.index.SegmentReader.get(SegmentReader.java:554)
	=
org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)
	=
org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRe=
ader.java:27)
	=
org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:75)
	=
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.jav=
a:677)
	=
org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
	org.apache.lucene.index.IndexReader.open(IndexReader.java:188)
	com.smartsheet.Main.main(<Unknown Source>:Unknown line)

-Mark



On Nov 17, 2010, at 1:51 PM, Michael McCandless wrote:

> Lucene interns field names... since you have a truly enormous number
> of unique fields it's expected intern will be called alot.
>=20
> But that said it's odd that it's this costly.
>=20
> Can you post the stack traces that call intern?
>=20
> Mike
>=20
> On Fri, Nov 5, 2010 at 1:53 PM, Michael McCandless
> <lucene@mikemccandless.com> wrote:
>> Hmm...
>>=20
>> So, I was going on this output from your CheckIndex:
>>=20
>>   test: field norms.........OK [296713 fields]
>>=20
>> But in fact I just looked and that number is bogus -- it's always
>> equal to total number of fields, not number of fields with norms
>> enabled.  I'll open an issue to fix this, but in the meantime can you
>> apply this patch to your CheckIndex and run it again?
>>=20
>> Index: src/java/org/apache/lucene/index/CheckIndex.java
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>> --- src/java/org/apache/lucene/index/CheckIndex.java    (revision =
1031678)
>> +++ src/java/org/apache/lucene/index/CheckIndex.java    (working =
copy)
>> @@ -570,8 +570,10 @@
>>       }
>>       final byte[] b =3D new byte[reader.maxDoc()];
>>       for (final String fieldName : fieldNames) {
>> -        reader.norms(fieldName, b, 0);
>> -        ++status.totFields;
>> +        if (reader.hasNorms(fieldName)) {
>> +          reader.norms(fieldName, b, 0);
>> +          ++status.totFields;
>> +        }
>>       }
>>=20
>>       msg("OK [" + status.totFields + " fields]");
>>=20
>> So if in fact you have already disabled norms then something else is
>> the source of the sudden slowness.  Though, such a huge number of
>> unique field names is not an area of Lucene that's very well =
tested...
>> perhaps there's something silly somewhere.  Maybe you can try
>> profiling just the init of your IndexReader?  (Eg, run java with
>> -agentlib:hprof=3Dcpu=3Dsamples,depth=3D16,interval=3D1).
>>=20
>> Yes, both Index.NOT_ANALYZED_NO_NORMS and Index.NO will disable norms
>> as long as no document in the index ever had norms on (yes it does
>> "infect" heh).
>>=20
>> Mike
>>=20
>> On Fri, Nov 5, 2010 at 1:37 PM, Mark Kristensson
>> <mark.kristensson@smartsheet.com> wrote:
>>> While most of our Lucene indexes are used for more traditional =
searching, this index in particular is used more like a reporting =
repository. Thus, we really do need to have that many fields indexed and =
they do need to be broken out into separate fields. There may be another =
way to structure the index to reduce the number of fields, but I'm =
hoping we can optimize the current design and avoid (yet another) index =
redesign.
>>>=20
>>> I'll look into the tweaking the merge policy, but I'm more =
interested in disabling norms because scoring really doesn't matter for =
us. Basically, we need nothing more than a binary answer from Lucene: =
either a record meets the provided criteria (which can be a rather =
complex boolean query with many subqueries) or it doesn't. If the record =
does match, then we get the IDs from lucene and run off to get the live =
data from our primary data store and sort it (in Java) based upon =
criteria provided by the user, not by score.
>>>=20
>>> After our initial design mushroomed in size, we redesigned and now =
(I thought) do not have norms on any of the fields in this index. So, =
I'm wondering if there was something in the results from the CheckIndex =
that I provided which indicates to you that we may have norms still =
enabled? I know that if you have norms on any one document's field, then =
any other document with that same field will get "infected" with norms =
as well.
>>>=20
>>> My understanding is that any field that uses the constants  =
Index.NOT_ANALYZED_NO_NORMS or  Index.NO will not  have norms on it, =
regardless of whether or not the field is stored. Is that not correct?
>>>=20
>>> Thanks,
>>> Mark
>>>=20
>>>=20
>>>=20
>>> On Nov 4, 2010, at 2:56 AM, Michael McCandless wrote:
>>>=20
>>>> Likely what happened is you had a bunch of smaller segments, and =
then
>>>> suddenly they got merged into that one big segment (_aiaz) in your
>>>> index.
>>>>=20
>>>> The representation for norms in particular is not sparse, so this
>>>> means the size of the norms file for a given segment will be
>>>> number-of-unique-indexed-fields X number-of-documents.
>>>>=20
>>>> So this count grows quadratically on merge.
>>>>=20
>>>> Do these fields really need to be indexed?   If so, it'd be better =
to
>>>> use a single field for all users for the indexable text if you can.
>>>>=20
>>>> Failing that, a simple workaround is to set the maxMergeMB/Docs on =
the
>>>> merge policy; this'd prevent big segments from being produced.
>>>> Disabling norms should also workaround this, though that will =
affect
>>>> hit scores...
>>>>=20
>>>> Mike
>>>>=20
>>>> On Wed, Nov 3, 2010 at 7:37 PM, Mark Kristensson
>>>> <mark.kristensson@smartsheet.com> wrote:
>>>>> Yes, we do have a large number of unique field names in that =
index, because they are driven by user named fields in our application =
(with some cleaning to remove illegal chars).
>>>>>=20
>>>>> This slowness problem has appeared very suddenly in the last =
couple of weeks and the number of unique field names has not spiked in =
the last few weeks. Have we crept over some threshold with our linear =
growth in the number of unique field names? Perhaps there is a limit =
driven by the amount of RAM in the machine that we are violating? Are =
there any guidelines for the maximum number, or suggested number, of =
unique fields names in an index or segment? Any suggestions for =
potentially mitigating the problem?
>>>>>=20
>>>>> Thanks,
>>>>> Mark
>>>>>=20
>>>>>=20
>>>>> On Nov 3, 2010, at 2:02 PM, Michael McCandless wrote:
>>>>>=20
>>>>>> On Wed, Nov 3, 2010 at 4:27 PM, Mark Kristensson
>>>>>> <mark.kristensson@smartsheet.com> wrote:
>>>>>>>=20
>>>>>>> I've run checkIndex against the index and the results are below. =
That net is that it's telling me nothing is wrong with the index.
>>>>>>=20
>>>>>> Thanks.
>>>>>>=20
>>>>>>> I did not have any instrumentation around the opening of the =
IndexSearcher (we don't use an IndexReader), just around the actual =
query execution so I had to add some additional logging. What I found =
surprised me, opening a search against this index takes the same 6 to 8 =
seconds that closing the indexWriter takes.
>>>>>>=20
>>>>>> IndexWriter opens a SegmentReader for each segment in the index, =
to
>>>>>> apply deletions, so I think this is the source of the slowness.
>>>>>>=20
>>>>>> =46rom the CheckIndex output, it looks like you have many =
(296,713)
>>>>>> unique fields names on that one large segment -- does that sound
>>>>>> right?  I suspect such a very high field count is the source of =
the
>>>>>> slowness...
>>>>>>=20
>>>>>> Mike
>>>>>>=20
>>>>>> =
---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>>=20
>>>>>=20
>>>>>=20
>>>>> =
---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>>=20
>>>>>=20
>>>>=20
>>>> =
---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>>=20
>>>=20
>>>=20
>>=20
>=20
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>=20


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org