lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <>
Subject Re: FST and FieldCache?
Date Thu, 19 May 2011 18:01:11 GMT
On Thu, May 19, 2011 at 20:43, Michael McCandless
<> wrote:
> On Thu, May 19, 2011 at 12:35 PM, Jason Rutherglen
> <> wrote:
>>> And I do agree there are times when mmap is appropriate, eg if query
>>> latency is unimportant to you, but it's not a panacea and it comes
>>> with serious downsides
>> Do we have a benchmark of ByteBuffer vs. byte[]'s in RAM?
> I don't know of a straight up comparison...
I did compare MMapDir vs RAMDir variant a couple of years ago.
Searches slowed down a teeny-weeny little bit. GC times went down
noticeably. For me it was a big win.

Whatever Mike might say, mmap is great for latency-conscious applications : )

If someone tries to create artificial benchmark for byte[] VS
ByteBuffer, I'd recommend going through Lucene's abstraction layer.
If you simply read/write in a loop, JIT will optimize away boundary
checks for byte[] in some cases. This didn't ever happen to *Buffer
family for me.

>> There's also RAM based SSDs whose performance could be comparable with
>> well, RAM.
> True, though it's through layers of abstraction designed originally
> for serving files off of spinning magnets :)
>> Also, with our heap based field caches, the first sorted
>> search requires that they be loaded into RAM.  Then we don't unload
>> them until the reader is closed?  With MMap the unloading would happen
>> automatically?
> True, but really if the app knows it won't need that FC entry for a
> long time (ie, long enough to make it worth unloading/reloading) then
> it should really unload it.  MMap would still have to write all those
> pages to disk...
> DocValues actually makes this a lot cheaper because loading DocValues
> is much (like ~100 X from Simon's testing) faster than populating
> FieldCache since FieldCache must do all the uninverting.
> Mike
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

Kirill Zakharenko/Кирилл Захаренко
Phone: +7 (495) 683-567-4
ICQ: 104465785

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message