lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "M A" <geneticfl...@googlemail.com>
Subject Re: Search Performance Problem 16 sec for 250K docs
Date Sun, 20 Aug 2006 12:56:44 GMT
Just ran some tests .. it appears that the problem is in the sorting ..

i.e.

//hits = searcher.search(query, new Sort("sid", true));    -> 17 secs
//hits = searcher.search(query, new Sort("sid", false)); -> 17 secs
hits = searcher.search(query);    -> less than 1 sec ..

am trying something out .. .. will keep you posted





On 8/20/06, Erik Hatcher <erik@ehatchersolutions.com> wrote:
>
> This is why a warming strategy like Solr takes is very valuable.  The
> searchable index is always serving up requests as fast as Lucene
> works, which is achieved by warming a new IndexSearcher with searches/
> sorts/filter creating/etc before it is swapped into use.
>
>        Erik
>
>
> On Aug 20, 2006, at 5:35 AM, M A wrote:
>
> > Ok I get your point, this still however means the first search on
> > the new
> > searcher will take a huge amount of time .. given that this is
> > happening now
> > ..
> >
> > i.e. new search -> new query -> get hits ->20+ secs ..  this
> > happens every 5
> > mins or so ..
> >
> > although subsequent searches may be quicker ..
> >
> > Am i to assume for a first search the amount of  time is ok -> ..
> > seems like
> > a long time to me ..?
> >
> > The other thing is the sorting is fixed .. it never changes .. it
> > is always
> > sorted by the same field ..
> >
> > i just built the entire index and it still takes ages .,..
> >
> >
> >
> >
> >
> >
> >
> >
> > On 8/20/06, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> >>
> >>
> >> : This is because the index is updated every 5 mins or so, due to the
> >> incoming
> >> : feed of stories ..
> >> :
> >> : When you say iteration, i take it you mean, search request, well
> >> for
> >> each
> >> : search that is conducted I create a new one .. search reader
> >> that is ..
> >>
> >> yeah ... i ment iteration of your test.  don't do that.
> >>
> >> if the index is updated every 5 minutes, then open a new searcher
> >> every 5
> >> minutes -- and reuse it for theentire 5 minutes.  if it's updated
> >> "sparadically throughout the day" then open a search, and keep
> >> using it
> >> untill the index is udated, then open a new one.
> >>
> >> reusing an indexsearcher as long as possible is one of biggest
> >> factors of
> >> Lucene applications.
> >>
> >> :
> >> :
> >> :
> >> : On 8/19/06, Chris Hostetter <hossman_lucene@fucit.org> wrote:
> >> : >
> >> : >
> >> : > :     hits = searcher.search(query, new Sort("sid", true));
> >> : >
> >> : > you don't show where searcher is initialized, and you don't
> >> clarify
> >> how
> >> : > you are timing your multiple iterations -- i'm going to guess
> >> that you
> >> are
> >> : > opening a new searcher every iteration right?
> >> : >
> >> : > sorting on a field requires pre-computing an array of
> >> information for
> >> that
> >> : > field -- this is both time and space expensive, and is cached per
> >> : > IndexReader/IndexSearcher -- so if you reuse the same searcher
> >> and
> >> time
> >> : > multiple iterations you'll find that hte first iteration might be
> >> somewhat
> >> : > slow, but the rest should be very fast.
> >> : >
> >> : >
> >> : >
> >> : > -Hoss
> >> : >
> >> : >
> >> : >
> >> ---------------------------------------------------------------------
> >> : > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> : > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> : >
> >> : >
> >> :
> >>
> >>
> >>
> >> -Hoss
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message