lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Hostetter <hossman_luc...@fucit.org>
Subject Re: Index Partitioning ( was Re: Search deadlocking under load)
Date Mon, 11 Jul 2005 00:43:36 GMT

: > Generally speaking, you only ever need one active Searcher, which
: > all of
: > your threads should be able to use.  (Of course, Nathan says that
: > in his
: > code base, doing this causes his JVM to freeze up, but I've never seen
: > this myself).
: >
: Thanks for your response Chris.  Do you think we are going down a
: deadly path by having "many smaller" IndexSearchers open rather than
: "one very large one"?

I'm sorry ... i think i may have confused you, i forgot  that this thread
was regarding partioning the index.  i ment one searcher *per index* ...
don't try to make a seperate searcher per client, or have a pool of
searchers, or anything like that.  But if you have a need to partition
your data into multiple indexes, then have one searcher per index.

I don't really know a lot about what gets loaded into memory when you
make/use a new searcher, but the one thing i've learned from experience is
that the FieldCache (which gets used when you sort on a field) contains
every term in the field you are sorting on, and an instance of FieldCache
exists for each IndexReader you open (which is one big reason not to open
a seperate reader for every client).

now assume you partition your data into two seperate indexes, unless the
way you partition your data lets you cleanly so that each of hte
two indexes contains only half the number of terms as if you had one big
index, then sorting on a field in those two indexes will require more RAM
then sorting on the same data in asingle index.

...that's just one example i ran into when looking at partitioning data,
i'm sure there are other cases where splitting your data up into seperate
indexes isn't neccessarily more efficient then using one big index.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message