Return-Path: 
 <java-user-return-16880-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-java-user-archive@www.apache.org
Received: (qmail 79596 invoked from network); 12 Oct 2005 14:10:26 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199)
  by minotaur.apache.org with SMTP; 12 Oct 2005 14:10:26 -0000
Received: (qmail 79939 invoked by uid 500); 12 Oct 2005 14:10:18 -0000
Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org
Received: (qmail 79842 invoked by uid 500); 12 Oct 2005 14:10:14 -0000
Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:java-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:java-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:java-user@lucene.apache.org>
List-Id: <java-user.lucene.apache.org>
Reply-To: java-user@lucene.apache.org
Delivered-To: mailing list java-user@lucene.apache.org
Received: (qmail 79810 invoked by uid 99); 12 Oct 2005 14:10:12 -0000
Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Oct 2005 07:10:12 -0700
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=HTML_MESSAGE,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (asf.osuosl.org: domain of yseeley@gmail.com designates
 72.14.204.197 as permitted sender)
Received: from [72.14.204.197] (HELO qproxy.gmail.com) (72.14.204.197)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Oct 2005 07:10:14 -0700
Received: by qproxy.gmail.com with SMTP id e12so206071qbe
        for <java-user@lucene.apache.org>;
 Wed, 12 Oct 2005 07:09:50 -0700 (PDT)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=o0DWBqG5qoBJtc2NJ2eLAYuYHDz9uwq1oTwkpw9pc3X7bufx+vI+LlrMFpfzFMOXWQ9+eXPN6/Wgx2wNBtTqPju9YRwsMS+kBpbVQ+iaA+m9Km4+Z6WZ0AGLsqRzthtLdKXojJEr6wSWwDSoNLDpthU3Ckurmf2faEgVUWfPc6Q=
Received: by 10.65.150.20 with SMTP id c20mr396176qbo;
        Wed, 12 Oct 2005 07:08:19 -0700 (PDT)
Received: by 10.65.121.10 with HTTP; Wed, 12 Oct 2005 07:08:19 -0700 (PDT)
Message-ID: <c68e39170510120708x39113cc9r269cc82176d5c76a@mail.gmail.com>
Date: Wed, 12 Oct 2005 10:08:19 -0400
From: Yonik Seeley <yseeley@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: "docMap" array in SegmentMergeInfo
In-Reply-To: <e994873a0510120519u299f596fk339957c7abf0de82@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_18666_8015088.1129126099498"
References: <20050713184257.55124.qmail@web41208.mail.yahoo.com>
	 <42D5881C.6040304@apache.org>
	 <e994873a0510111514s4051c52fp397cd5e42cedad22@mail.gmail.com>
	 <88c6a6720510111623m3e09dbb2y4605c6c4a7485bf3@mail.gmail.com>
	 <c68e39170510112040j6ed0f2buad6a02ed01d75ce9@mail.gmail.com>
	 <e994873a0510120519u299f596fk339957c7abf0de82@mail.gmail.com>
X-Virus-Checked: Checked by ClamAV on apache.org
X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N

------=_Part_18666_8015088.1129126099498
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

Thanks for the trace Peter, and great catch!
It certainly does look like avoiding the construction of the docMap for a
MultiTermEnum will be a significant optimization.


-Yonik
Now hiring -- http://tinyurl.com/7m67g

On 10/12/05, Peter Keegan <peterlkeegan@gmail.com> wrote:
>
> Here is one stack trace:
>
> Full thread dump Java HotSpot(TM) Client VM (1.5.0_03-b07 mixed mode):
>
> "Thread-6" prio=3D5 tid=3D0x6cf7a7f0 nid=3D0x59e50 waiting for monitor en=
try
> [0x6d2cf000..0x6d2cfd6c]
> at org.apache.lucene.index.SegmentReader.isDeleted(SegmentReader.java:241=
)
> - waiting to lock <0x04e40278> (a org.apache.lucene.index.SegmentReader)
> at org.apache.lucene.index.SegmentMergeInfo.<init>(SegmentMergeInfo.java
> :43)
> at org.apache.lucene.index.MultiTermEnum.<init>(MultiReader.java:277)
> at org.apache.lucene.index.MultiReader.terms(MultiReader.java:186)
> at org.apache.lucene.search.RangeQuery.rewrite(RangeQuery.java:75)
> at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:243)
> at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:166)
> at org.apache.lucene.search.Query.weight(Query.java:84)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:158)
> at org.apache.lucene.search.Searcher.search(Searcher.java:67)
> at org.apache.lucene.search.QueryFilter.bits(QueryFilter.java:62)
> at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:121)
> at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:64)
> at org.apache.lucene.search.Hits.<init>(Hits.java:51)
> at org.apache.lucene.search.Searcher.search(Searcher.java:49)
>
> I've also seen it happen during sorting from:
>
> FieldSortedHitQueue.comparatorAuto ->
> FieldCacheImpl.getAuto() ->
> MultiReader.terms() ->
> MultiTermEnum.init() ->
> SegmentMergerInfo.init() ->
> SegmentReader.isDeleted()
>
> Peter
>
> On 10/11/05, Yonik Seeley <yseeley@gmail.com> wrote:
> >
> > > We've been using this in production for a while and it fixed the
> > > extremely slow searches when there are deleted documents.
> >
> > Who was the caller of isDeleted()? There may be an opportunity for an
> easy
> > optimization to grab the BitVector and reuse it instead of repeatedly
> > calling isDeleted() on the IndexReader.
> >
> > -Yonik
> > Now hiring -- http://tinyurl.com/7m67g
> >
>

------=_Part_18666_8015088.1129126099498--