lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Analyzing performance and memory consumption for boolean queries
Date Wed, 24 Jun 2009 22:26:33 GMT
On Wed, Jun 24, 2009 at 3:38 PM, Nigel<nigelspleen@gmail.com> wrote:

> Yes, we're indexing on a separate server, and rsyncing from index snapshots
> there to the search servers.  Usually rsync has to copy just a few small
> .cfs files, but every once in a while merging will product a big one.  I'm
> going to try to limit this by setting maxMergeMB, but of course that's a
> trade-off with having more segments.

OK.

> It sounds like surely any swapping out of the JVM memory could cause big and
> unpredictable performance drops.  As I just mentioned in reply to Uwe, our
> poor performance times don't always directly correlate with index updates,
> but it may be that the damage is done and the effects are only seen sometime
> later.

OK... I wonder whether the bytes written by rsync go into the IO
cache.  I would assume they do.  But in your case that might be OK
since presumably you'll then cutover to those new files, so, they've
been pre-warmed by rsync.

> We don't store norms (since we don't care about sort order), and we don't
> have any deleted docs (since the index is read-only on the search servers).

Ahh excellent.

> What exactly is stored in the field cache?

FieldCache is used when sorting by field (not relevance) and by
function queries, or, if you directly load values, eg
FieldCache.DEFAULT.getInts(...).

> p.s. I haven't tried SSDs yet, or for that matter faster disks of any sort.
> First I'd like to get a better understanding of what I/O is required and
> when during the search process, ideally to be able to have an approximate
> model that predicts I/O based on the query (the way a DBA might do when
> estimating how a SQL query would work with certain tables and indexes).

Sounds like a good plan!!  And, after that, upgrade ;)  There's no
going back once you make the switch...

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message