lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aalap Parikh <alo...@yahoo.com>
Subject Re: Index Partitioning ( was Re: Search deadlocking under load)
Date Mon, 11 Jul 2005 19:08:37 GMT
>I don't really know a lot about what gets loaded into
memory when you
>make/use a new searcher, but the one thing i've
learned from experience 
>is
>that the FieldCache (which gets used when you sort on
a field) contains
>every term in the field you are sorting on, and an
instance of 
>FieldCache
>exists for each IndexReader you open (which is one
big reason not to 
>open
>a seperate reader for every client).

You mentioned that re-using the same IndexSearcher
would provide better performance in terms of sorting
of search results, but what if the index I am
searching on is constantly being updated? Would using
the same Searcher pick up those updates (adds/removes)
or that I would need to instantiate a new Searcher for
each of the client requests in order for the search
results to reflect the updated or new docs in the
index?

Thanks,
Aalap.

--- Chris Hostetter <hossman_lucene@fucit.org> wrote:

> 
> : > Generally speaking, you only ever need one
> active Searcher, which
> : > all of
> : > your threads should be able to use.  (Of course,
> Nathan says that
> : > in his
> : > code base, doing this causes his JVM to freeze
> up, but I've never seen
> : > this myself).
> : >
> : Thanks for your response Chris.  Do you think we
> are going down a
> : deadly path by having "many smaller"
> IndexSearchers open rather than
> : "one very large one"?
> 
> I'm sorry ... i think i may have confused you, i
> forgot  that this thread
> was regarding partioning the index.  i ment one
> searcher *per index* ...
> don't try to make a seperate searcher per client, or
> have a pool of
> searchers, or anything like that.  But if you have a
> need to partition
> your data into multiple indexes, then have one
> searcher per index.
> 
> I don't really know a lot about what gets loaded
> into memory when you
> make/use a new searcher, but the one thing i've
> learned from experience is
> that the FieldCache (which gets used when you sort
> on a field) contains
> every term in the field you are sorting on, and an
> instance of FieldCache
> exists for each IndexReader you open (which is one
> big reason not to open
> a seperate reader for every client).
> 
> now assume you partition your data into two seperate
> indexes, unless the
> way you partition your data lets you cleanly so that
> each of hte
> two indexes contains only half the number of terms
> as if you had one big
> index, then sorting on a field in those two indexes
> will require more RAM
> then sorting on the same data in asingle index.
> 
> ...that's just one example i ran into when looking
> at partitioning data,
> i'm sure there are other cases where splitting your
> data up into seperate
> indexes isn't neccessarily more efficient then using
> one big index.
> 
> 
> 
> -Hoss
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail:
> java-user-help@lucene.apache.org
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message