lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis_gospodne...@yahoo.com>
Subject Re: PATCH: SegmentsReader/SegmentsTermEnum
Date Wed, 10 Sep 2003 14:17:09 GMT
Christoph,

Thanks for the patch and the test.
I refactored your test a bit, converted it to a JUnit-based unit test
and will commit it shortly, following it with your patch.

Thank you,
Otis

--- Christoph Goller <goller@detego-software.de> wrote:
> Hi Lucene Developers,
> 
> first let me thank you all for this excellent peace of software
> that you created. I am using Lucene in several projects and I
> am currently also building more enhanced text mining applications
> on top of it. Because of that I have spent a lot of time studying
> the Lucene sources and I will come up with a couple of proposals
> for bug fixes in the next days. Here is the first one:
> 
> I think I can fix a bug in SegmentsTermEnum.
> One can create a TermEnum from an IndexReader in two ways:
> 
> indexReader.terms()
> indexReader.terms(t)
> 
> If one gets a TermEnum starting at a specified term t one does not
> have to call enum.next() before using it. The enum is valid from the
> beginning.Calling enum.next() switches to the next term. However,
> this
> bahaviour is only true if our index consists of only one segment. If
> we
> have an index consisting of several segments term t is delivered
> twice,
> 1st time after calling indexReader.terms(t); enum.term(), 2nd time
> after
> calling enum.next(). Furthermore the initial document frequency might
> be false (if t occurs in more than one segment). The problem can be
> fixed by calling next() in the constructor of SegmentsTermEnum.
> I attach a test that demonstrates the problem and a patch that fixes
> it.
> 
> kind regards,
> Christoph
> 
> -- 
> *****************************************************************
> * Dr. Christoph Goller       Tel.:   +49 89 203 45734           *
> * Detego Software GmbH       Mobile: +49 179 1128469            *
> * Keuslinstr. 13             Fax.:   +49 721 151516176          *
> * 80798 München, Germany     Email:  goller@detego-software.de  *
> *****************************************************************
> > import java.io.IOException;
> 
> import org.apache.lucene.analysis.WhitespaceAnalyzer;
> import org.apache.lucene.document.Document;
> import org.apache.lucene.document.Field;
> import org.apache.lucene.index.IndexReader;
> import org.apache.lucene.index.IndexWriter;
> import org.apache.lucene.index.Term;
> import org.apache.lucene.index.TermEnum;
> import org.apache.lucene.store.Directory;
> import org.apache.lucene.store.RAMDirectory;
> 
> /*
>  * Created on 23.04.2003
>  *
>  * To change the template for this generated file go to
>  * Window>Preferences>Java>Code Generation>Code and Comments
>  */
> 
> /**
>  * @author goller
>  *
>  * To change the template for this generated type comment go to
>  * Window>Preferences>Java>Code Generation>Code and Comments
>  */
> public class SegmentsTermEnumTest {
> 
>   int docCount = 0;
>   
>   void addDoc1(IndexWriter writer)
>   {
>     Document doc = new Document();
>     
>     doc.add(Field.Keyword("id","id" + docCount));
>     doc.add(Field.UnStored("content","aaa"));
>     
>     try {
>       writer.addDocument(doc);
>     }
>     catch (IOException e) {
>       // TODO Auto-generated catch block
>       e.printStackTrace();
>     }
>     docCount++;
>   }
>   
>   void addDoc2(IndexWriter writer)
>   {
>     Document doc = new Document();
>     
>     doc.add(Field.Keyword("id","id" + docCount));
>     doc.add(Field.UnStored("content","aaa bbb"));
>     
>     try {
>       writer.addDocument(doc);
>     }
>     catch (IOException e) {
>       // TODO Auto-generated catch block
>       e.printStackTrace();
>     }
>     docCount++;
>   }
>   
>   
>   
>   public static void main(String[] args) 
>   {
>     //System.out.println(System.getProperty("java.version"));
>     
>     Directory dir = new RAMDirectory();
>     SegmentsTermEnumTest test = new SegmentsTermEnumTest();
>     
>     IndexWriter writer = null;
>     IndexReader reader = null;
>     TermEnum enum = null;
>     int i;
>     
>     try {
>       writer  = new IndexWriter(dir, new WhitespaceAnalyzer(), true);
>       
>       for (i = 0; i < 100; i++)
>         test.addDoc1(writer);
>       
>       for (i = 0; i < 100; i++)
>         test.addDoc2(writer);
>       
>       writer.close();
>     }
>     catch (IOException e) {
>       // TODO Auto-generated catch block
>       e.printStackTrace();
>     }
>     
>     
>     try {
>       reader = IndexReader.open(dir);
>   
>       System.out.println("terms():");
>       enum = reader.terms();
>       for(i = 0; i < 5 && enum.next(); i++)
>         System.out.println(enum.term() + " " + enum.docFreq());
>       
>       enum.close();
>       
>       System.out.println();
>       System.out.println("terms(\"aaa\")");
>       enum = reader.terms(new Term("content", "aaa"));
>       System.out.println(enum.term() + " " + enum.docFreq());
>       for(i = 0; i < 5 && enum.next(); i++)
>         System.out.println(enum.term() + " " + enum.docFreq());
>         
>       enum.close();
>       reader.close();
>       
>       writer  = new IndexWriter(dir, new WhitespaceAnalyzer(),
> false);
>       writer.optimize();
>       writer.close();
>    
>       System.out.println();
>       System.out.println("optimize");
>       
>       reader = IndexReader.open(dir);
>   
>       System.out.println();
>       System.out.println("terms():");
>       enum = reader.terms();
>       for(i = 0; i < 5 && enum.next(); i++)
>         System.out.println(enum.term() + " " + enum.docFreq());
>       
>       enum.close();
>       
>       System.out.println();
>       System.out.println("terms(\"aaa\")");
>       enum = reader.terms(new Term("content", "aaa"));
>       System.out.println(enum.term() + " " + enum.docFreq());
>       for(i = 0; i < 5 && enum.next(); i++)
>         System.out.println(enum.term() + " " + enum.docFreq());
>         
>       enum.close();
>       reader.close();
>       
>     }
>     catch (IOException e2) {
>       // TODO Auto-generated catch block
>       e2.printStackTrace();
>     }
>     
> 
>     
>     
>     
>     
>   }
> }
> > Index: SegmentsReader.java
> ===================================================================
> RCS file:
>
/home/cvspublic/jakarta-lucene/src/java/org/apache/lucene/index/SegmentsReader.java,v
> retrieving revision 1.11
> diff -u -r1.11 SegmentsReader.java
> --- SegmentsReader.java	1 May 2003 01:09:15 -0000	1.11
> +++ SegmentsReader.java	3 Sep 2003 13:03:27 -0000
> @@ -238,9 +238,7 @@
>      }
>  
>      if (t != null && queue.size() > 0) {
> -      SegmentMergeInfo top = (SegmentMergeInfo)queue.top();
> -      term = top.termEnum.term();
> -      docFreq = top.termEnum.docFreq();
> +      next();
>      }
>    }
>  
> 
> >
---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-dev-help@jakarta.apache.org


__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

Mime
View raw message