Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of zhoucheng2008@gmail.com
 designates 209.85.215.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAL8PwkbeVA5TH521+ZEriuzbAC5kC1jii+ncnLFDaA+PN-QzCA@mail.gmail.com>
References: 
 <CA+huq0D7YNYsiHhgTY25xEiREE2JvBpDvL3jbw1Sj+xPUW3sdA@mail.gmail.com>
	<F899CDA770CA4B2BBA01BE7AE4CA68E6@JackKrupansky>
	<CA+huq0AcoeDiZE9frMTPy2dPwpH68d=5nMYjtbu2H31WVaPE+A@mail.gmail.com>
	<CACn_-AxP94OBYZ594HiQWzfZkFrx5jR_eNKMeUVThJE0tydi9g@mail.gmail.com>
	<CA+huq0CEMoh+wwbv-MM6dFUU=mGOmE+Z9X0t6KY6px6bECsfLg@mail.gmail.com>
	<885dea4a-bfd8-4c22-8c02-6b926d9e0f30@email.android.com>
	<CA+huq0BKx4g66QKKP5DBOT7QFCmimQ8YEfCVnUuwM+D-BRHcTA@mail.gmail.com>
	<017101cd42ad$30be5d90$923b18b0$@thetaphi.de>
	<CA+huq0Aw_25EhLJJqBx2fcjChZ91w0Vjx1HA1BPiTTjsvnovQA@mail.gmail.com>
	<CAL8PwkbeVA5TH521+ZEriuzbAC5kC1jii+ncnLFDaA+PN-QzCA@mail.gmail.com>
Date: Mon, 18 Jun 2012 19:42:35 +0800
Message-ID: 
 <CA+huq0AOe-FyuCWPsXwVs5g5sdMfCXe6=UrphqTEx8hqTEkbtw@mail.gmail.com>
Subject: Re: RAMDirectory unexpectedly slows
From: Cheng <zhoucheng2008@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=bcaec555547e81314604c2bda964

--bcaec555547e81314604c2bda964
Content-Type: text/plain; charset=ISO-8859-1

Lucene is used in the following steps:

1) store interested data in Lucene indexes;

2) search key words against the indexes;

3) write new data into indexes and refresh the reader;

4) use the reader to search key words, and the 2-4 steps repeat.

As you see, there are lots of read, update actions. I guess since MMapDir
needs to synchronize to a local drive that causes it to be slower.


The code is attached:

public class YYTLucene {

private static Logger logger = Logger.getLogger(YYTLuceneImpl.class);

private static FSDirectory indexDir;

private static RAMDirectory ramDir;

// private static MMapDirectory ramDir;

private static IndexWriter iw;

private static IndexSearcher is;

private static IndexReader ir;

private static YYTLucene instance;

public static YYTLucene getInstance(String type) {
if (instance == null) {
instance = new YYTLucene(type);
}
return instance;
}

private YYTLucene(String type) {
try {
indexDir = new NIOFSDirectory(new File(ERConstants.indexFolder1
+ "/" + type));

ramDir = new RAMDirectory(indexDir);

// ramDir = new MMapDirectory(new File(ERConstants.indexFolder1 +
// "/"
// + type));

IndexWriterConfig iwConfig = new IndexWriterConfig(
ERConstants.version, new LimitTokenCountAnalyzer(
ERConstants.analyzer, ERConstants.maxTokenNum));

// iwConfig.setMaxBufferedDocs(ERConstants.maxBufferedDocs);
//
// iwConfig.setRAMBufferSizeMB(ERConstants.RAMBufferSizeMB);

iw = new IndexWriter(ramDir, iwConfig);
iw.commit();

ir = IndexReader.open(iw, true);
is = new IndexSearcher(ir);

} catch (IOException e) {
e.printStackTrace();
logger.info("Can't initiate YYTLuceneImpl...");
}
}

public IndexWriter getIndexWriter() {
return iw;
}

public void setIndexWriter(IndexWriter iw) {
YYTLucene.iw = iw;
}

public IndexSearcher getIndexSearcher() {
return is;
}

public void setIndexSearcher(IndexSearcher is) {
YYTLucene.is = is;
}

public IndexReader getIndexReader() {
return ir;
}

public static void setIndexReader(IndexReader ir) {
YYTLucene.ir = ir;
}

}


On Mon, Jun 18, 2012 at 7:32 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> 9 fold improvement using RAMDir over MMapDir is much more than I've
> seen (~30-40% maybe) in the past.
>
> Can you explain how you are using Lucene?
>
> You may also want to try the CachingRAMDirectory patch on
> https://issues.apache.org/jira/browse/LUCENE-4123
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sat, Jun 16, 2012 at 7:18 AM, Cheng <zhoucheng2008@gmail.com> wrote:
> > After a number of test, the performance of MMapDirectory is not even
> close
> > to that of RAMDirectory, in terms of speed.
> >
> > My application w/ the former can only deal with 10 tasks per round while
> it
> > could handle over 90 w/ RAMDirectory.
> >
> > I use the application in Linux.
> >
> > What can be the reasons?
> >
> > Thanks.
> >
> >
> > On Tue, Jun 5, 2012 at 7:53 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
> >
> >> This is managed by your operating system. In general OS kernels like
> Linux
> >> or Windows use all free memory to cache disk accesses.
> >>
> >> -----
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: uwe@thetaphi.de
> >>
> >>
> >> > -----Original Message-----
> >> > From: Cheng [mailto:zhoucheng2008@gmail.com]
> >> > Sent: Monday, June 04, 2012 6:10 PM
> >> > To: java-user@lucene.apache.org
> >> > Subject: Re: RAMDirectory unexpectedly slows
> >> >
> >> > Can I control the size of ram given to either MMapDirectory or
> >> > ByteBufferDirectory?
> >> >
> >> > On Mon, Jun 4, 2012 at 11:42 PM, Uwe Schindler <uwe@thetaphi.de>
> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > If you are using MMapDirectory or this ByteBufferDirectory (which is
> >> > > similar to the first) the used RAM is outside JVM heap, it is in the
> >> > > FS cache of the OS kernel. Giving too much memory to the JVM
> penalizes
> >> > > the OS cache, so give only as much as the App needs. Lucene and the
> OS
> >> > > kernel will then utilize the remaining memory for caching.
> >> > >
> >> > > Please read docs of MMapDirectory and inform yourself about mmap in
> >> e.g.
> >> > > Wikipedia.
> >> > >
> >> > > Uwe
> >> > > --
> >> > > Uwe Schindler
> >> > > H.-H.-Meier-Allee 63, 28213 Bremen
> >> > > http://www.thetaphi.de
> >> > >
> >> > >
> >> > >
> >> > > Cheng <zhoucheng2008@gmail.com> schrieb:
> >> > >
> >> > > Please shed more insight into the difference between JVM heap size
> and
> >> > > the memory size used by Lucene.
> >> > >
> >> > > What I am getting at is that no matter however much ram I give my
> >> > > apps, Lucene can't utilize it. Is that right?
> >> > >
> >> > > What about the ByteBufferDirectory? Can this specific directory
> >> > > utilize the 2GB memory I grant to the app?
> >> > >
> >> > > On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> >> > > jason.rutherglen@gmail.com> wrote:
> >> > >
> >> > > > If you want the index to be stored completely in RAM, there is the
> >> > > > ByteBuffer directory [1]. Though I do not see the point in putting
> >> > > > an index in RAM, it will be cached in RAM regardless in the OS
> >> > > > system IO cache.
> >> > > >
> >> > > > 1.
> >> > > >
> >> > >
> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/ja
> >> > > va/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
> >> > > >
> >> > > > On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zhoucheng2008@gmail.com>
> >> > wrote:
> >> > > > > My indexes are 500MB+. So it seems like that RAMDirectory is not
> >> > > > > good
> >> > > for
> >> > > > > that big a size.
> >> > > > >
> >> > > > > My challenge, on the other side, is that I need to update the
> >> > > > > indexes
> >> > > > very
> >> > > > > frequently. So, do you think MMapDirectory is the solution?
> >> > > > >
> >> > > > > Thanks.
> >> > > > >
> >> > > > > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <
> >> > > jack@basetechnology.com
> >> > > > >wrote:
> >> > > > >
> >> > > > >> From the javadoc for RAMDirectory:
> >> > > > >>
> >> > > > >> "Warning: This class is not intended to work with huge indexes.
> >> > > > Everything
> >> > > > >> beyond several hundred megabytes will waste resources (GC
> >> > > > >> cycles),
> >> > > > because
> >> > > > >> it uses an internal buffer size of 1024 bytes, producing
> millions
> >> > > > >> of byte[1024] arrays. This class is optimized for small
> >> > > > >> memory-resident indexes. It also has bad concurrency on
> >> multithreaded
> >> > environments.
> >> > > > >>
> >> > > > >> It is recommended to materialize large indexes on disk and use
> >> > > > >> MMapDirectory, which is a high-performance directory
> >> > > > >> implementation
> >> > > > working
> >> > > > >> directly on the file system cache of the operating system, so
> >> > > > >> copying
> >> > > > data
> >> > > > >> to Java heap space is not useful."
> >> > > > >>
> >> > > > >> -- Jack Krupansky
> >> > > > >>
> >> > > > >> -----Original Message----- From: Cheng
> >> > > > >> Sent: Monday, June 04, 2012 10:08 AM
> >> > > > >> To: java-user@lucene.apache.org
> >> > > > >> Subject: RAMDirectory unexpectedly slows
> >> > > > >>
> >> > > > >>
> >> > > > >> Hi,
> >> > > > >>
> >> > > > >> My apps need to read from and write to some big indexes
> >> frequently.
> >> > > So I
> >> > > > >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB
> >> > > > >> memory size.
> >> > > > >>
> >> > > > >> I notice that the speed of reading and writing unexpectedly
> slows
> >> > > > >> as
> >> > > the
> >> > > > >> size of the indexes increases. Since the usage of RAM is less
> >> > > > >> than
> >> > > 20%,
> >> > > > I
> >> > > > >> think by default the RAMDirectory doesn't take advantage of the
> >> > > memory I
> >> > > > >> assigned to JVM.
> >> > > > >>
> >> > > > >> What are the steps to improve the reading and writing speed of
> >> > > > >> RAMDirectory?
> >> > > > >>
> >> > > > >> Thanks!
> >> > > > >> Jeff
> >> > > > >>
> >> > > > >>
> >> > > >_____________________________________________
> >> > > **_____________________________________________
> >> > > **---------
> >> > > > >> To unsubscribe, e-mail:
> >> > > > >> java-user-unsubscribe@lucene.**apache.org<
> >> > > > java-user-unsubscribe@lucene.apache.org>
> >> > > > >> For additional commands, e-mail:
> >> > > > >> java-user-help@lucene.apache.**org<
> >> > > > java-user-help@lucene.apache.org>
> >> > > > >>
> >> > > > >>
> >> > > >
> >> > > >_____________________________________________
> >> > >
> >> > > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > > > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> > > >
> >> > > >
> >> > >
> >> > >
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--bcaec555547e81314604c2bda964--