lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Collins <chris_j_coll...@yahoo.com>
Subject Re: Optimizing indexes with mulitiple processors?
Date Fri, 10 Jun 2005 20:31:38 GMT
Yeh I think the bug is related to an array copy that expects 1k blocks (if I
recall it was RAMDirectory or something like that).  

C

--- Kevin Burton <burton@rojo.com> wrote:

> Chris Collins wrote:
> 
> >Well I am currently looking at merging too.  In my application merging will
> >occur against a filer (read as higher latency device).  I am currently
> working
> >on how to stage indices on local disk before moving to a filer.  Assume I
> must
> >move to a filer eventually for whatever crazzy reason I need to....dont ask
> it
> >aint funny :-}
> >
> >In that case I have a different performance issue, that is that
> FSInputStream
> >and FSOutputStream inherit the buffer size of 1k from OS and IS  This would
> be
> >useful to increase to reduce the amount of RPC's to the filer when doing
> merges
> >..... assuming that reads and writes are sequential (CIFS supports a 64k
> block
> >and NFS supports upto I think 32k). 
> >
> Yeah.. I already did this actually ... on local disks the performance 
> benefit wasn't noticable.  The variables are  private/final ... I made 
> them public and non-final and it worked.
> 
> Note that OutputStream has a bug when I set it higher... I don't have 
> the trace I'm afraid...
> 
> > I haven't spent much time on this so far
> >so its not like I know its hard todo.  From preliminary experiments its
> obvious
> >that changing the OS buffersize is not the thing todo. 
> >
> >If anyone has successfully increased the FSOutputStream and FSInputStream
> >buffers and got it not to blow up on array copies I would love to know the
> >short cut
> >
> Maybe that was my problem...
> 
> Kevin
> 
> -- 
> 
> 
> Use Rojo (RSS/Atom aggregator)! - visit http://rojo.com. 
> See irc.freenode.net #rojo if you want to chat.
> 
> Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
> 
>    Kevin A. Burton, Location - San Francisco, CA
>       AIM/YIM - sfburtonator,  Web - http://peerfear.org/
> GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 




---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message