lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: IO bandwidth throttling
Date Tue, 02 Aug 2005 21:55:02 GMT
Hi,

> Ok, let me rephrase the question. Assuming the RAMDirectory holds
> approximately 500 MB of data which needs to be written to the
> filesystem, I'm afraid that sending this much data in one shot might
> choke the NFS. Is there a parameter with FSDirectory with which I can
> instruct Lucene to restrict the amount of data my application writes
> to the filesystem per second.

There is no support for that, but if you add it, this may be a nice
enhancement.

> BTW, I could see maxMergeDocs, but not maxBufferedDocs with the
> IndexWriter. Otis, is there really some parameter with that name? I
> think maxMergeDocs may not work well here because the size of the
> documents are pretty random, ranging from few hundred bytes to tens
> of megabytes.

That's the one.   You are looking at 1.4.3, and I was referring to the
new name of that parameter from 1.9-dev1.

Otis

> On 8/3/05, Otis Gospodnetic <otis_gospodnetic@yahoo.com> wrote:
> > While not exactly what you are describing, you can use one of the
> > IndexWriter parameters (maxBufferedDocs) to control the size of the
> > RAMDirectory that's used as a buffer during indexing.
> > 
> > Otis
> > 
> > --- Gopikrishnan Subramani <gopi.subramani@gmail.com> wrote:
> > 
> > > Hello,
> > >
> > > Is there a way I can control the IO bandwidth utilized by Lucene?
> > >
> > > Here is my scenario. RAMDirectory is used to build a in-memory
> index
> > > and finally the index size approaches a limit, the contents are
> > > flushed to a FSDirectory. The index size could be approximately
> 512
> > > MB. I'm a bit concerned that this might affect other NFS clients
> of
> > > the same NFS server, even if it's for a short period. To avoid
> this I
> > > would like to implement some sort of limit on the amount of data
> > > transferred to the filesystem akin to the --bwlimit with the
> rsync
> > > command.
> > >
> > > Thanks,
> > > Gopi
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> > 
> > 
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> > 
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message