Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of uwe@thetaphi.de designates
 188.138.97.18 as permitted sender)
From: "Uwe Schindler" <uwe@thetaphi.de>
To: <java-user@lucene.apache.org>
References: 
 <CANm_Fuea_fD5h1ShJ=WctoH6D7COtJpcMsL6aFYZi84dbXUSHw@mail.gmail.com>
	<B9B46467-9353-4D49-AD58-08250B9701C5@likeness.com>
	<CAN4YXvdp=tNC62S_3a6398cr-Sh1S_iaGSBTRukZXfRNOzL7CQ@mail.gmail.com>
	<CANm_FufC=C=MGbnJe=DP4dRGP6_541Jk4F9gCQyev0M=kHUFGg@mail.gmail.com>
	<51D1F76F.9020702@gmail.com>
 <CADMUOEWqgtEpk8H-678-QnBO7xjACwEBNooW+tTKrfERZzehJw@mail.gmail.com>
In-Reply-To: 
 <CADMUOEWqgtEpk8H-678-QnBO7xjACwEBNooW+tTKrfERZzehJw@mail.gmail.com>
Subject: RE: In memory index (current status in Lucene)
Date: Sat, 6 Jul 2013 21:59:45 +0200
Message-ID: <022901ce7a83$5d16e8d0$1744ba70$@thetaphi.de>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Thread-Index: 
 AQG1NFY5Nwy1ZWLwUgsJ8YiK20yToQI2COt3AUx1OuMClX7LBQJI367RAcyeFZyZOTH7cA==
Content-Language: de

You mean tmpfs - not RAM disk. Tmpfs is cool, as it plays wonderful =
winth mmap (mmap just maps the RAM used by the tmpfs into the user's =
address space).

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Ramkumar R. Aiyengar [mailto:andyetitmoves@gmail.com]
> Sent: Thursday, July 04, 2013 10:14 PM
> To: java-user@lucene.apache.org
> Subject: Re: In memory index (current status in Lucene)
>=20
> Have you tried using MMapDirectory over a RAM disk (assuming you are =
on
> Linux)? You can avoid writing to disk (and thus the other ways to get =
to it
> persistently as Steven mentions), but still MMap it.
> On 1 Jul 2013 22:41, "Lance Norskog" <goksron@gmail.com> wrote:
>=20
> > My current open source project is a Directory that is just like
> > RAMDirectory, but everything is memory-mapped. The idea is it =
creates
> > a disk file, opens it, and immediately deletes the file. The file
> > still exists until the IndexReader/Writer/Searcher closes it. But, =
it
> > cannot be found from the file system. This is just like a
> > RAMDirectory, but without memory limitations.
> >
> > It's proving to be harder than it looked.
> >
> > The application is to store encrypted indexes in memory, with the
> > decrypted contents in this non-findable format. I'm in medical
> > document analysis now, and we can't store anything on disk in the =
clear.
> >
> > Lance
> >
> > On 07/01/2013 07:07 AM, Emmanuel Espina wrote:
> >
> >> Hi Erick! Nice to hear from you again! From time to time my =
interest
> >> in these "Lucene things" returns and I do some experiments :p
> >>
> >> Just to add to this conversation, I found an interesting link to
> >> Mike's blog about memory resident indexes (using another virtual
> >> machine) http://blog.mikemccandless.**com/2012/07/lucene-index-in-
> **
> >> ram-with-azuls-zing-
> jvm.html<http://blog.mikemccandless.com/2012/07/l
> >> ucene-index-in-ram-with-azuls-zing-jvm.html>
> >> and also (which is not exactly what I asked but seems related) =
there
> >> is a Google Summer of Code project to build a memory residen term
> >> resident:
> >> http://www.google-melange.com/**gsoc/project/google/gsoc2013/**
> >> billybob/42001<http://www.google-
> melange.com/gsoc/project/google/gsoc
> >> 2013/billybob/42001>
> >>
> >> Thanks
> >> Emmanuel
> >>
> >>
> >> 2013/7/1 Erick Erickson <erickerickson@gmail.com>:
> >>
> >>> Hey Emma! It's been a while....
> >>>
> >>> Building on what Steven said, here's Uwe's blog on MMapDirectory =
and
> >>> Lucene:
> >>> http://blog.thetaphi.de/2012/**07/use-lucenes-mmapdirectory-**
> >>> on-64bit.html<http://blog.thetaphi.de/2012/07/use-lucenes-
> mmapdirect
> >>> ory-on-64bit.html>
> >>>
> >>> I've always considered RAMDirectory for rather restricted =
use-cases.
> >>> I.e. if I know without doubt that the index is both relatively
> >>> static and bounded. The other use I've seen is to use it to index
> >>> single documents on-the-fly for some reason (say complex =
processing
> >>> of a single result) then throw it out afterwards.
> >>>
> >>> How are things going?
> >>>
> >>> Erick
> >>>
> >>>
> >>>
> >>> On Fri, Jun 28, 2013 at 5:36 PM, Steven Schlansker
> >>> <steven@likeness.com
> >>> >wrote:
> >>>
> >>>  On Jun 28, 2013, at 2:29 PM, Emmanuel Espina
> >>> <espinaemmanuel@gmail.com>
> >>>> wrote:
> >>>>
> >>>>  I'm building a distributed index (mostly as a reasearch project
> >>>> for
> >>>>> school) and I'm evaluating indexing the entire collection in
> >>>>> memory (like google, facebook and others have done years ago). =
The
> >>>>> obvious reason for this is performance considering that the
> >>>>> replication will give me a reasonably good durability of the =
data
> >>>>> (despite being in volatile memory).
> >>>>>
> >>>>> What is the current status of Lucene for this kind of indexes?
> >>>>> RAMDirectory in it's documentation has a scary warning that says
> >>>>> that "is not intended to work with huge indexes", and that =
sounds
> >>>>> more like it is an implementation for testing rather than
> >>>>> something for production.
> >>>>>
> >>>>> Of course there is no real context for this question, because it
> >>>>> is a reasearch topic. Testing it's limits would be the closest =
to
> >>>>> a context I have :p
> >>>>>
> >>>> You could consider MMapDirectory, which will end up putting the
> >>>> active portions of the index in memory (via the filesystem buffer
> >>>> cache).
> >>>>
> >>>> The benefit is that you don't completely destroy the Java heap
> >>>> (RAMDirectory causes immense GC pressure if you are not careful)
> >>>> and you don't have to commit all of your ram to index usage all =
the
> >>>> time.
> >>>>
> >>>> The downside is that if your working set exceeds the amount of =
RAM
> >>>> available for buffer cache, you will get silent performance
> >>>> degradation as you fall back to disk reads for the missing =
blocks.
> >>>>
> >>>> Maybe this is OK for your use case, maybe not.
> >>>>
> >>>>
> >>>> ------------------------------**------------------------------**
> >>>> ---------
> >>>> To unsubscribe, e-mail:
> >>>> java-user-unsubscribe@lucene.**apache.org<java-user-
> unsubscribe@luc
> >>>> ene.apache.org> For additional commands, e-mail:
> >>>> java-user-help@lucene.apache.**org<java-user-
> help@lucene.apache.org
> >>>> >
> >>>>
> >>>>
> >>>>  ------------------------------**------------------------------**
> >> ---------
> >> To unsubscribe, e-mail:
> >> java-user-unsubscribe@lucene.**apache.org<java-user-
> unsubscribe@lucen
> >> e.apache.org> For additional commands, e-mail:
> >> java-user-help@lucene.apache.**org<java-user-
> help@lucene.apache.org>
> >>
> >>
> >
> > =
------------------------------**------------------------------**------
> > --- To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.**apache.org<java-user-
> unsubscribe@lucene
> > .apache.org> For additional commands, e-mail:
> > java-user-help@lucene.apache.**org<java-user-
> help@lucene.apache.org>
> >
> >


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org