lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From WolfgangTäger <wtae...@epo.org>
Subject Performance problem
Date Wed, 24 Aug 2005 07:32:08 GMT
Dear all,

we are using Lucene to store 10Mio bilingual sentence pairs for doing some 
natural language processing with them. Each documents contains a sentence, 
its translation and a topical code. We want to select sentences containing 
certain words and do statistics over the topical codes in order to detect 
translations which depend on the topic (like key=> Taste (topic: input 
devices), key=> Schlüssel (topic: cryptography)).

While the search is carried out in a reasonably short time (about 
500..800ms) we have a performance problem with actually retrieving the 
documents by code like:

for (int i = nrhits-1; i >=0; i--){
        Document hitDoc = hits.doc(i);
        String code=hitDoc.get("code");
        ... statistics
}
 
Even when restricting nrhits to 2000, we have to wait 10..20 seconds just 
for the retrieval. Since the documents are so short we would have expected 
a quicker retrieval. BtW the loop was done in inverse order in the hope to 
accelerate the retrieval.

We are using Lucene 1.4.3 Java version on a Windows PC.
 
Would you recommend using the C version ? I suppose it is stable and we 
can reuse the database ? Any other suggestions ?

Thanks for your help !

Wolfgang

 
 
 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message