lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl Austin" <Carl.aus...@detica.com>
Subject RE: FieldCache and 2.9
Date Tue, 11 May 2010 13:47:20 GMT
Ah, ok, thanks for that. 
I had hoped that the field cache would do this for me, by going through the subreaders itself.
Is this likely to be done in a future release?
I may have to implement some wrapper that does this anyway, and if so, can submit it as contrib
if that would be useful?

Thanks

Carl

-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
Sent: 11 May 2010 14:41
To: java-user@lucene.apache.org
Subject: Re: FieldCache and 2.9

You are requesting the FieldCache entry from the top-level reader and hence a whole new FieldCache
entry must be created.
Lucene 2.9 sorting requests FieldCache entries at the segment level and hence reuses entries
for those segments that haven't changed.

-Yonik
Apache Lucene Eurocon 2010
18-21 May 2010 | Prague



On Tue, May 11, 2010 at 9:27 AM, Carl Austin <Carl.austin@detica.com> wrote:
> Hi,
>
> I have been using the FieldCache in lucene version 2.9 compared to 
> that in 2.4. The load time is massively decreased, however I am not 
> seeing any benefit in getting a field cache after re-open of an index 
> reader when I have only added a few extra documents.
> A small test class is included below (based off one from Lucid 
> Imagination), that creates 5Mil docs, gets a field cache, creates 
> another few docs and gets the field cache again. I though the second 
> get would be very very fast, as only 1 segment should have changed, 
> however it takes more time for the reopen and cache get than it did 
> the original.
>
> Am I doing something wrong here or have I misunderstood the new 
> segment changes?
>
> Thanks
>
> Carl
>
>
> import java.io.File;
>
> import org.apache.lucene.analysis.SimpleAnalyzer;
> import org.apache.lucene.document.Document;
> import org.apache.lucene.document.Field; import 
> org.apache.lucene.index.IndexReader;
> import org.apache.lucene.index.IndexWriter;
> import org.apache.lucene.search.FieldCache;
> import org.apache.lucene.store.Directory;
> import org.apache.lucene.store.FSDirectory;
>
> public class ContrivedFCTest {
>
>        public static void main(String[] args) throws Exception {
>                Directory dir = FSDirectory.open(new File(args[0]));
>                IndexWriter writer = new IndexWriter(dir, new 
> SimpleAnalyzer(), true,
>                                IndexWriter.MaxFieldLength.LIMITED);
>                for (int i = 0; i < 5000000; i++) {
>                        if (i % 100000 == 0) {
>                                System.out.println(i);
>                        }
>                        Document doc = new Document();
>                        doc.add(new Field("field", "String" + i, 
> Field.Store.NO,
>                                        Field.Index.NOT_ANALYZED));
>                        writer.addDocument(doc);
>                }
>                writer.close();
>
>                IndexReader reader = IndexReader.open(dir, true);
>                long start = System.currentTimeMillis();
>                FieldCache.DEFAULT.getStrings(reader, "field");
>                long end = System.currentTimeMillis();
>                System.out.println("load time for initial field cache:"
> + (end - start)
>                                / 1000.0f + "s");
>
>                writer = new IndexWriter(dir, new SimpleAnalyzer(), 
> false,
>                                IndexWriter.MaxFieldLength.LIMITED);
>                for (int i = 5000001; i < 5000005; i++) {
>                        if (i % 100000 == 0) {
>                                System.out.println(i);
>                        }
>                        Document doc = new Document();
>                        doc.add(new Field("field", "String" + i, 
> Field.Store.NO,
>                                        Field.Index.NOT_ANALYZED));
>                        writer.addDocument(doc);
>                }
>                writer.close();
>
>                IndexReader reader2 = reader.reopen(true);
>                System.out.println("reader size = " + 
> reader2.numDocs());
>                long start2 = System.currentTimeMillis();
>                FieldCache.DEFAULT.getStrings(reader2, "field");
>                long end2 = System.currentTimeMillis();
>                System.out.println("load time for re-opened field 
> cache:"
>                                + (end2 - start2) / 1000.0f + "s");
>        }
> }
>
> This message should be regarded as confidential. If you have received this email in error
please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy by an authorised
signatory.  The contents of this email may relate to dealings with other companies within
the Detica Limited group of companies.
>
> Detica Limited is registered in England under No: 1337451.
>
> Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

This message should be regarded as confidential. If you have received this email in error
please notify the sender and destroy it immediately.
Statements of intent shall only become binding when confirmed in hard copy by an authorised
signatory.  The contents of this email may relate to dealings with other companies within
the Detica Limited group of companies.

Detica Limited is registered in England under No: 1337451.

Registered offices: Surrey Research Park, Guildford, Surrey, GU2 7YP, England.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message