lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Nikolic <stefan.niko...@gmail.com>
Subject Re: Using multiple drives and non-CFS format to improve search performance
Date Fri, 27 Aug 2010 15:26:40 GMT
Sanne,

I'd prefer to use symlinks for existing, static indexes, but I'm really glad
you showed me FileSwitchDirectory. I think it will be useful for me in the
future. Thanks for your advice!

Any tips on how to convert an existing CFS index to non-CFS? I'm aware of
the extractor in IndexReader, but I don't know how to repair/rebuild the
segments_N file afterwards. Is this even possible?

Also, I found this:
http://lucene.472066.n3.nabble.com/Distributing-index-over-N-disks-td577483.html
Gives
me hope!

Thanks again,
Stefan

On Fri, Aug 27, 2010 at 2:35 AM, Sanne Grinovero
<sanne.grinovero@gmail.com>wrote:

> Hi Stefan,
> you might want to consider org.apache.lucene.store.FileSwitchDirectory
> before going for the symlinks approach.
> Sorry I don't know the effect nor recommended file types, I would
> naively start setting the smallest on SSD, then perform tests, but
> that's possibly not the best scenario:
> under stress the most-frequently used files will likely be cached by
> your operating system, so to achieve the perfect setup you would need
> to find which is the most frequently used types of files, excluding
> what fits in your os cache.. I assume it's going to change with
> configurations, index sizes and use cases: not an easy task.
>
> Sanne
>
> 2010/8/26 Stefan Nikolic <stefan.nikolic@gmail.com>:
> > Hi everyone,
> >
> > I'm trying to figure out the effects on search performance of using the
> > non-CFS format and spreading the various underlying files to different
> > disks/media types. For example, I'm considering moving a segment's
> various
> > .t* term-related files onto a solid-state drive, the .fdx/.fdt
> > stored-fields-related files onto a standard rotational drive, and using
> > symlinks to hide all of this from Lucene.
> >
> > Any ideas on what the performance effects of such a setup would be? Which
> > files would you recommend putting on the slower media, and which on the
> > faster media?
> >
> > Thanks!
> >
> > -Stefan
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message