lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dawid Weiss <dawid.we...@cs.put.poznan.pl>
Subject Re: Storing and loading the FST directly from disk
Date Fri, 03 Jun 2011 06:19:36 GMT
> In any case I browsed the code a bit more, it'd be easy to test out
> the MMap'ing because the FST simply reads directly from a byte[],
> which could be converted to a ByteBuffer (that's MMap'd).  Nice!

The FST is basically a byte[], but like Robert said -- the layout of
transitions and states will require nearly random access patterns all
around it. There are some optimizations one can make to lay out
initial nodes close to each other, for example (and I've done it as
part of another project) -- these do optimize CPU cache utilization
and I would assume MMAP'ped buffers as well -- but once you go past
page fault zone the performance will be terrible. And I assume the
degradation will not be smooth, but very sharp.

This said, I'd be interested in looking at the benchmarks if you want
to invest some time and do it ;)

Dawid

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message