lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Distributing index over N disks
Date Wed, 25 Nov 2009 09:58:53 GMT
I think this is a good idea, for indexes that can't fit in IO cache.
Report back if you get good results :)  I think FSD opens up all sorts
of interesting possibilities.

Mike

On Tue, Nov 24, 2009 at 5:31 PM, Otis Gospodnetic
<otis_gospodnetic@yahoo.com> wrote:
> Hello,
>
> Would it make sense and be possible to spread different index files over multiple disks
(without resorting to putting an index on a RAID)?
> For example, what if the index files didn't live in a single index dir, but were organized
by their type in a snallow dir tree, like this:
>
> /path/to/index:
>   tis/<tis files here>
>   ftd/<fdt files here>
>   prx/<prx files here>
>   ...
>
> Then one could symlink these tis, fdt, prx, etc. dirs to locations that are really on
different disks.
> Is this doable and would it help imrpve performance?  I think it could improve segment
merging, index optimization, and searches, because N disk heads would be able to do ~N times
more work because of parallelization.
>
>
> But the idea seems to simple that it makes me think I'm missing something, otherwise
it would have already been done. :)
>
> Otis
> --
> Sematext is hiring -- http://sematext.com/about/jobs.html?mls
> Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message