lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Collins <chris_j_coll...@yahoo.com>
Subject Re: Optimizing indexes with mulitiple processors?
Date Fri, 10 Jun 2005 06:52:56 GMT
Well I am currently looking at merging too.  In my application merging will
occur against a filer (read as higher latency device).  I am currently working
on how to stage indices on local disk before moving to a filer.  Assume I must
move to a filer eventually for whatever crazzy reason I need to....dont ask it
aint funny :-}

In that case I have a different performance issue, that is that FSInputStream
and FSOutputStream inherit the buffer size of 1k from OS and IS  This would be
useful to increase to reduce the amount of RPC's to the filer when doing merges
..... assuming that reads and writes are sequential (CIFS supports a 64k block
and NFS supports upto I think 32k).  I haven't spent much time on this so far
so its not like I know its hard todo.  From preliminary experiments its obvious
that changing the OS buffersize is not the thing todo. 

If anyone has successfully increased the FSOutputStream and FSInputStream
buffers and got it not to blow up on array copies I would love to know the
short cut.

Chris

--- Kevin Burton <burton@rojo.com> wrote:

> Chris Collins wrote:
> 
> >To follow up.  I was surprised to find that from the experiment of indexing
> 4k
> >documents to local disk (Dell PE with onboard RAID with 256MB cache). I got
> the
> >following data from my profile:
> >
> >70 % time was spent in inverting the document
> >30 % in merge
> >
> >  
> >
> Oh.. yeah.. thats indexing.  I'm more interested in merging multiple 
> indexes...
> 
> Kevin
> 
> -- 
> 
> 
> Use Rojo (RSS/Atom aggregator)! - visit http://rojo.com. 
> See irc.freenode.net #rojo if you want to chat.
> 
> Rojo is Hiring! - http://www.rojonetworks.com/JobsAtRojo.html
> 
>    Kevin A. Burton, Location - San Francisco, CA
>       AIM/YIM - sfburtonator,  Web - http://peerfear.org/
> GPG fingerprint: 5FB2 F3E2 760E 70A8 6174 D393 E84D 8D04 99F1 4412 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message