Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:Received:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=32jhDTRA9RhuqdYf41p48PHK7mDN809ijOtXvW0T6LxmGnynTZHd0cgckNockSNUVgiEkiKFDBdWPtG7NSBuAr85JqvP8pbytWZekukVQdRDqgdg1rnp2aFctBvSDh4IzcJ9QUFnKtczQgfkdsVWVfDIBfDJWieJjeekizlVHwI=
  ;
Message-ID: <20060721185750.37039.qmail@web50313.mail.yahoo.com>
Date: Fri, 21 Jul 2006 11:57:50 -0700 (PDT)
From: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Reply-To: Otis Gospodnetic <otis_gospodnetic@yahoo.com>
Subject: Re: Fastest Method for Searching (need all results)
To: java-user@lucene.apache.org
In-Reply-To: <E2306077-A633-4CEF-B366-95620F857942@genome.chop.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

I haven't had the chance to use this new feature yet, but have you tried with selective field loading, so that you can load only that 1 field from your index and not all of them?

Otis

----- Original Message ----
From: Ryan O'Hara <ohara@genome.chop.edu>
To: java-user@lucene.apache.org
Sent: Friday, July 21, 2006 2:43:41 PM
Subject: Fastest Method for Searching (need all results)

My index contains approximately 5 millions documents.  During a  
search, I need to grab the value of a field for every document in the  
result set.  I am currently using a HitCollector to search.  Below is  
my code:

searcher.search(query, new HitCollector(){
                         public void collect(int doc, float score){
                                 if(searcher.doc(doc).get("SYM") !=  
null){
                                     addSymbolsToHash(searcher.doc 
(doc).get("SYM").split("ENDOFSYM"));
                                 }
                         }
                     });

This is fairly fast for small and medium-sized result sets.  However,  
it gets slow as the result set grows.  I read this on HitCollector's  
API page:

"For good search performance, implementations of this method should  
not call Searcher.doc(int) or Reader.document(int) on every document  
number encountered. Doing so can slow searches by an order of  
magnitude or more."

Along with this implementation, I've also tried using FieldCache.   
This faired better with large-sized result sets, but worse with small  
and medium-sized result sets.  Anyone have any ideas of what the best  
approach might be?

Thanks a lot,
Ryan


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org