From java-user-return-24897-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org Thu Dec 07 18:31:37 2006
Return-Path: <java-user-return-24897-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-java-user-archive@www.apache.org
Received: (qmail 79834 invoked from network); 7 Dec 2006 18:31:36 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Dec 2006 18:31:36 -0000
Received: (qmail 71501 invoked by uid 500); 7 Dec 2006 18:31:36 -0000
Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org
Received: (qmail 71478 invoked by uid 500); 7 Dec 2006 18:31:36 -0000
Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:java-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:java-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:java-user@lucene.apache.org>
List-Id: <java-user.lucene.apache.org>
Reply-To: java-user@lucene.apache.org
Delivered-To: mailing list java-user@lucene.apache.org
Received: (qmail 71467 invoked by uid 99); 7 Dec 2006 18:31:36 -0000
Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Dec 2006 10:31:36 -0800
X-ASF-Spam-Status: No, hits=4.0 required=10.0
	tests=HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (herse.apache.org: domain of erickerickson@gmail.com designates 64.233.182.188 as permitted sender)
Received: from [64.233.182.188] (HELO nf-out-0910.google.com) (64.233.182.188)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Dec 2006 10:31:26 -0800
Received: by nf-out-0910.google.com with SMTP id n28so862251nfc
        for <java-user@lucene.apache.org>; Thu, 07 Dec 2006 10:31:05 -0800 (PST)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=RsGgzPCS/lQjdkt+7xK+V/5yId2b+SjRP9RCeH2cl2H/7lxT+Qg962x0YOOHlSXm7jiU7S0SbUH+iWncLjyR1dIcFVFDE87ZgJB/YhL3PCSRC0RRW1uFmp/X9pH37MklHRF3OfJ3kVjvjNHdeL9NFcfB3mVkNg5d9IFxk/+uUuw=
Received: by 10.82.179.9 with SMTP id b9mr107359buf.1165516264569;
        Thu, 07 Dec 2006 10:31:04 -0800 (PST)
Received: by 10.82.162.9 with HTTP; Thu, 7 Dec 2006 10:31:04 -0800 (PST)
Message-ID: <359a92830612071031g3fecf777x5751f61ec6c891d7@mail.gmail.com>
Date: Thu, 7 Dec 2006 13:31:04 -0500
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Reading Performance
In-Reply-To: <6B6D3063DF0A7A4DB8A989D40C61BEB3027FF0EC@EMAIL02.wescodist.com>
MIME-Version: 1.0
Content-Type: multipart/alternative; 
	boundary="----=_Part_66384_19856485.1165516264541"
References: <4578353F.2080206@gmail.com>
	 <6B6D3063DF0A7A4DB8A989D40C61BEB3027FF0EC@EMAIL02.wescodist.com>
X-Virus-Checked: Checked by ClamAV on apache.org

------=_Part_66384_19856485.1165516264541
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Well, the performance isn't bad considering you're executing the *search*
around 1,000 times.......

One of the characteristics of a Hits object is that it's optimized for
getting the top 100 docs or so. To get the next 100 docs it re-executes the
query. Repeatedly <G>. I'd try using a HitCollector or TopDocs instead of a
Hits object....

Hope this helps
Erick

On 12/7/06, Aigner, Thomas <TAigner@wescodist.com> wrote:
>
>
>
>
>
> Howdy all,
>
>
>
>       I have a question on reading many documents and time to do this.
> I have a loop on the hits object reading a record, then writing it to a
> file.  When there is only 1 user on the Index Searcher, this process to
> read say 100,000 takes around 3 seconds.  This is slow, but can be
> acceptable.  When a few more users do searchers, this time to just read
> from the hits object becomes well over 10 seconds, sometimes even 30+
> seconds.  Is there a better way to read through and do something with
> the hits information?  And yes, I have to read all of them to do this
> particular task.
>
>
>
> for (int i = 0;(i <= hits.length() - 1); i++)
>
> {
>
>
>
>       if (fw == null)
>
>       {
>
>             fw = new BufferedWriter( new FileWriter( searchWriteSpec ),
> 8196) ;
>
>       }
>
>
>
>       //Write Out records
>
>       String tmpHold = "";
>
> tmpHold = hits.doc(i).get("somefield1") + hits.doc(i).get("somefield2");
>
>
>
>       fw.write(tmpHold + "\n" );
>
>
>
> }
>
>
>
> Any ideas on how to speed this up especially with multiple users?  Each
> user gets their own class which has the above code in it.
>
>
>
> Thanks,
>
> Tom
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

------=_Part_66384_19856485.1165516264541--