lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Conway" <gcon...@textwise.com>
Subject lucene applicability and performance
Date Wed, 28 Apr 2004 19:44:53 GMT
Hello.  Apologies if this has come up before, I'm new to the list and
didn't see anything in the archives that exactly matched my situation.

I am considering using Lucene to index and search a large collection of
small documents in a  specialized domain -- probably only a few
thousands unique terms spanning across anywhere from one million to ten
million small source documents.  I hope to be able to get ranked search
results back in less than 400 msec.

I suspect one issue I may face is index density owing to the large
numbers of documents and relatively small vocabulary.  That, in turn,
may be a drag on query processing.  I am working on strategies to
ameliorate that somewhat but it may be difficult.

In the meantime, I'm looking for some gut reactions from the experts
before I take this to the next stage.  Can Lucene scale well to this
kind of situation?  Can I realistically hope to get anywhere near my
performance targets?  Will I have to distribute pieces of the index
across several machines,  parallelize my retrievals, and merge the
results to do so?  If so, does Lucene already support that or will I
have to develop that logic in house?  (Seems like I saw a reference
somewhere that such a feature was coming soon, but I'm not sure when or
how it will be implemented.)

Any help, tips, references, or advice would be welcome and appreciated.
Thank you!

Regards,

Greg 


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message