lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Configure writer to write to FSDirectory?
Date Mon, 06 Feb 2012 12:17:01 GMT
If you can use NRTManager and SearcherManager things should be easy
and blazingly fast rather than unbearably slow.  The latter phrase is
not one often associated with lucene.

IndexWriter iw = new IndexWriter(whatever - some standard disk index);
NRTManager nrtm = new NRTManager(iw, null);
NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, ...);
ropt.setXxx(...);
...
ropt.start();

SearcherManager srchm = nrtm.getSearcherManager(b);

Then add docs to your index via nrtm.addDocument(d), update with
nrtm.updateDocument(...), and to search use

IndexSearcher searcher = srchm.acquire();
try {
  search ...
} finally {
 srchm.release(searcher);
}

All thread safe so you don't have to worry about any complications
there.  And I bet it'll be blindingly fast.

Don't forget to close() things down at the end.


--
Ian.



On Mon, Feb 6, 2012 at 12:15 AM, Cheng <zhoucheng2008@gmail.com> wrote:
> I was trying to, but don't know how to even I read some of your blogs.
>
> On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
>> Are you using near-real-time readers?
>>
>> (IndexReader.open(IndexWriter))
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>> On Sun, Feb 5, 2012 at 9:03 AM, Cheng <zhoucheng2008@gmail.com> wrote:
>> > Hi Uwe,
>> >
>> > My challenge is that I need to update/modify the indexes frequently while
>> > providing the search capability. I was trying to use FSDirectory, but
>> found
>> > out that the reading and writing from/to FSDirectory is unbearably slow.
>> So
>> > I now am trying the RAMDirectory, which is fast.
>> >
>> > I don't know of  MMapDirectory, and wonder if it is as fast as
>> RAMDirectory.
>> >
>> >
>> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
>> >
>> >> Hi Cheng,
>> >>
>> >> It seems that you use a RAMDirectory for *caching*, otherwise it makes
>> no
>> >> sense to write changes back. In recent Lucene versions, this is not a
>> good
>> >> idea, especially for large indexes (RAMDirectory eats your heap space,
>> >> allocates millions of small byte[] arrays,...). If you need something
>> like
>> >> a
>> >> caching Directory and you are working on a 64bit platform, you can use
>> >> MMapDirectory (where the operating system kernel manages the read/write
>> >> between disk an memory). MMapDirectory is returned by default for
>> >> FSDirectory.open() on most 64 bit platforms. The good thing: the
>> "caching"
>> >> space is outside your JVM heap, so does not slowdown the garbage
>> collector.
>> >> So be sure to *not* allocate too much heap space (-Xmx) to your search
>> app,
>> >> only the minimum needed to execute it and leave the rest of your RAM
>> >> available for the OS kernel to manage FS cache.
>> >>
>> >> Uwe
>> >>
>> >> -----
>> >> Uwe Schindler
>> >> H.-H.-Meier-Allee 63, D-28213 Bremen
>> >> http://www.thetaphi.de
>> >> eMail: uwe@thetaphi.de
>> >>
>> >>
>> >> > -----Original Message-----
>> >> > From: Cheng [mailto:zhoucheng2008@gmail.com]
>> >> > Sent: Sunday, February 05, 2012 7:56 AM
>> >> > To: java-user@lucene.apache.org
>> >> > Subject: Configure writer to write to FSDirectory?
>> >> >
>> >> > Hi,
>> >> >
>> >> > I build an RAMDirectory on a FSDirectory, and would like the writer
>> >> associated
>> >> > with the RAMDirectory to periodically write to hard drive.
>> >> >
>> >> > Is this achievable?
>> >> >
>> >> > Thanks.
>> >>
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> >> For additional commands, e-mail: java-user-help@lucene.apache.org
>> >>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message