lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravikumar Govindarajan <ravikumar.govindara...@gmail.com>
Subject Re: fadvise/madvise during segment-merges....
Date Wed, 21 May 2014 12:20:28 GMT
Great blog and lucid explanation

I think things have changed in recent kernel versions. I am no expert, but
could see some code related to this here
http://lxr.free-electrons.com/source/mm/fadvise.c?v=3.14

O_DIRECT will be terrible drag no? Will a battery-backed disk cache help
here?

We are using a SortingMergePolicy which most-often hits data randomly. Will
SEQUENTIAL help here?

Any reasons why you think DONTNEED will be less-useful?

--
Ravi


On Wed, May 21, 2014 at 2:49 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> You're right, segment merges can be catastrophic to ongoing searches.
> I explored the problem here:
> http://blog.mikemccandless.com/2010/06/lucene-and-fadvisemadvise.html
> but a lot has changed since then...
>
> SEQUENTIAL is probably best (if the OS implements it; I think the
> Linux kernel has improved here), because then the OS knows to read
> ahead and aggressively free the page once we are done using it.
>
> There is also O_DIRECT (e.g., using NativeUnixDirectory) for direct IO
> to bypass the buffer cache entirely.
>
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Wed, May 21, 2014 at 4:09 AM, Ravikumar Govindarajan
> <ravikumar.govindarajan@gmail.com> wrote:
> > Is it a good idea to use FADVISE_DONTNEED/MADVISE_DONTNEED flags during
> > segment merge reads?
> >
> > Buffer-Cache contains critical data belonging to searches. A
> segment-merge
> > has the potential to disturb the cache no?
> >
> > --
> > Ravi
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message