lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earwin Burrfoot <ear...@gmail.com>
Subject Re: FST and FieldCache?
Date Thu, 19 May 2011 18:01:11 GMT
On Thu, May 19, 2011 at 20:43, Michael McCandless
<lucene@mikemccandless.com> wrote:
> On Thu, May 19, 2011 at 12:35 PM, Jason Rutherglen
> <jason.rutherglen@gmail.com> wrote:
>>> And I do agree there are times when mmap is appropriate, eg if query
>>> latency is unimportant to you, but it's not a panacea and it comes
>>> with serious downsides
>>
>> Do we have a benchmark of ByteBuffer vs. byte[]'s in RAM?
>
> I don't know of a straight up comparison...
I did compare MMapDir vs RAMDir variant a couple of years ago.
Searches slowed down a teeny-weeny little bit. GC times went down
noticeably. For me it was a big win.

Whatever Mike might say, mmap is great for latency-conscious applications : )

If someone tries to create artificial benchmark for byte[] VS
ByteBuffer, I'd recommend going through Lucene's abstraction layer.
If you simply read/write in a loop, JIT will optimize away boundary
checks for byte[] in some cases. This didn't ever happen to *Buffer
family for me.

>> There's also RAM based SSDs whose performance could be comparable with
>> well, RAM.
>
> True, though it's through layers of abstraction designed originally
> for serving files off of spinning magnets :)
>
>> Also, with our heap based field caches, the first sorted
>> search requires that they be loaded into RAM.  Then we don't unload
>> them until the reader is closed?  With MMap the unloading would happen
>> automatically?
>
> True, but really if the app knows it won't need that FC entry for a
> long time (ie, long enough to make it worth unloading/reloading) then
> it should really unload it.  MMap would still have to write all those
> pages to disk...
>
> DocValues actually makes this a lot cheaper because loading DocValues
> is much (like ~100 X from Simon's testing) faster than populating
> FieldCache since FieldCache must do all the uninverting.
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>



-- 
Kirill Zakharenko/Кирилл Захаренко
E-Mail/Jabber: earwin@gmail.com
Phone: +7 (495) 683-567-4
ICQ: 104465785

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message