lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevin Burton <bur...@rojo.com>
Subject Re: Optimizing indexes with mulitiple processors?
Date Fri, 10 Jun 2005 17:51:04 GMT
Chris Collins wrote:

>Well I am currently looking at merging too.  In my application merging will
>occur against a filer (read as higher latency device).  I am currently working
>on how to stage indices on local disk before moving to a filer.  Assume I must
>move to a filer eventually for whatever crazzy reason I need to....dont ask it
>aint funny :-}
>
>In that case I have a different performance issue, that is that FSInputStream
>and FSOutputStream inherit the buffer size of 1k from OS and IS  This would be
>useful to increase to reduce the amount of RPC's to the filer when doing merges
>..... assuming that reads and writes are sequential (CIFS supports a 64k block
>and NFS supports upto I think 32k). 
>
Yeah.. I already did this actually ... on local disks the performance 
benefit wasn't noticable.  The variables are  private/final ... I made 
them public and non-final and it worked.

Note that OutputStream has a bug when I set it higher... I don't have 
the trace I'm afraid...

> I haven't spent much time on this so far
>so its not like I know its hard todo.  From preliminary experiments its obvious
>that changing the OS buffersize is not the thing todo. 
>
>If anyone has successfully increased the FSOutputStream and FSInputStream
>buffers and got it not to blow up on array copies I would love to know the
>short cut
>
Maybe that was my problem...

Kevin

-- 


Use Rojo (RSS/Atom aggregator)! - visit http://rojo.com. 
See irc.freenode.net #rojo if you want to chat.

Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html

   Kevin A. Burton, Location - San Francisco, CA
      AIM/YIM - sfburtonator,  Web - http://peerfear.org/
GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message