lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: FST and FieldCache?
Date Thu, 19 May 2011 15:59:48 GMT
On Thu, May 19, 2011 at 10:09 AM, Jason Rutherglen
<jason.rutherglen@gmail.com> wrote:
>> When
>> you mmap them you let the OS decide when to swap stuff out which mean
>> you pick up potentially high query latency waiting for these pages to
>> swap back in
>
> Right, however if one is using lets say SSDs, and the query time is
> less important, then MMap'ing would be fine.  Also it prevents deadly
> OOMs in favor of basic 'slowness' of the query.  If there is no
> performance degradation I think MMap'ing is a great option.  A common
> use case is an index that's far too large for a given server will
> simply not work today, whereas with MMap'ed field caches the query
> would complete, just extremely slowly.  If the user wishes to improve
> performance it's easy enough to add more hardware.

Well, be careful: if you just don't have enough memory to accomodate
all the RAM data structures Lucene needs... you're gonna be in trouble
with mmap too.  True, you won't hit OOMEs anymore, but instead you'll
be in a swap fest and your app is nearly unusable.

SSDs, while orders of magnitude faster than spinning magnets, are
still orders of magnitude slower than RAM.

But, yes, they obviously help substantially.  It's a one-way door...
you'll never go back once you've switched to SSDs.

And I do agree there are times when mmap is appropriate, eg if query
latency is unimportant to you, but it's not a panacea and it comes
with serious downsides.

I wish I could have the opposite of mmap from Java -- the ability to
pin the pages that hold important data structures.

Mike

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message