Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of grant.ingersoll@gmail.com
 designates 72.14.246.248 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:mime-version:in-reply-to:references:content-type:message-id:content-transfer-encoding:from:subject:date:to:x-mailer;
        b=HG1ci5vf25zWDDeFKIlzbYAEVpRynG1b9UVFi/jQ/AdBMznsMxVcKJzcxpTToTgESprnNRLBYwxDpUo62z3D2KorLy2R3QjTQKOUWSYQyiS7aivXK5YGJS+nbaTZFWe3tKWeB0j10cUhPWzQoEXh+evPyhHfL+at1Si2/1gT3jk=
Mime-Version: 1.0 (Apple Message framework v752.2)
In-Reply-To: <6B6D3063DF0A7A4DB8A989D40C61BEB3027FF0EC@EMAIL02.wescodist.com>
References: <6B6D3063DF0A7A4DB8A989D40C61BEB3027FF0EC@EMAIL02.wescodist.com>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <AAC285CA-BA0F-4EFD-AE80-1C24F7D3AA0E@gmail.com>
Content-Transfer-Encoding: 7bit
From: Grant Ingersoll <grant.ingersoll@gmail.com>
Subject: Re: Reading Performance
Date: Thu, 7 Dec 2006 13:23:50 -0500
To: java-user@lucene.apache.org

Have you done any profiling to identify hotspots in Lucene versus  
your application?

You might look into the FieldSelector code (used in IndexReader) in  
the Trunk version of Lucene could be used to only load the fields you  
are interested when getting the document from disk.  This can be  
useful if you have large fields that are being loaded that you don't  
necessarily need (thus skipping them).

Also, do you need the BufferedWriter construction and check in side  
the loop?  Probably small in comparison to loading, but  It seems  
like it is only created once, why have it in the loop?


On Dec 7, 2006, at 1:14 PM, Aigner, Thomas wrote:

>
>
>
>
> Howdy all,
>
>
>
>       I have a question on reading many documents and time to do this.
> I have a loop on the hits object reading a record, then writing it  
> to a
> file.  When there is only 1 user on the Index Searcher, this  
> process to
> read say 100,000 takes around 3 seconds.  This is slow, but can be
> acceptable.  When a few more users do searchers, this time to just  
> read
> from the hits object becomes well over 10 seconds, sometimes even 30+
> seconds.  Is there a better way to read through and do something with
> the hits information?  And yes, I have to read all of them to do this
> particular task.
>
>
>
> for (int i = 0;(i <= hits.length() - 1); i++)
>
> {
>
>
>
>       if (fw == null)
>
>       {
>
>             fw = new BufferedWriter( new FileWriter 
> ( searchWriteSpec ),
> 8196) ;
>
>       }
>
>
>
>       //Write Out records
>
>       String tmpHold = "";
>
> tmpHold = hits.doc(i).get("somefield1") + hits.doc(i).get 
> ("somefield2");
>
>
>
>       fw.write(tmpHold + "\n" );
>
>
>
> }
>
>
>
> Any ideas on how to speed this up especially with multiple users?   
> Each
> user gets their own class which has the above code in it.
>
>
>
> Thanks,
>
> Tom
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

------------------------------------------------------
Grant Ingersoll
http://www.grantingersoll.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org