lucene-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From busc...@apache.org
Subject svn commit: r1058718 [2/18] - in /lucene/dev/branches/realtime_search: ./ lucene/ lucene/contrib/ lucene/contrib/ant/src/java/org/apache/lucene/ant/ lucene/contrib/ant/src/test/org/apache/lucene/ant/ lucene/contrib/benchmark/ lucene/contrib/demo/src/ja...
Date Thu, 13 Jan 2011 19:53:39 GMT

Modified: lucene/dev/branches/realtime_search/lucene/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/CHANGES.txt?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/CHANGES.txt (original)
+++ lucene/dev/branches/realtime_search/lucene/CHANGES.txt Thu Jan 13 19:53:21 2011
@@ -119,6 +119,25 @@ Changes in backwards compatibility polic
   need to change it (e.g. using "\\" to escape '\' itself).  
   (Sunil Kamath, Terry Yang via Robert Muir)
 
+* LUCENE-2771: IndexReader.norms() now throws UOE on non-atomic IndexReaders. If
+  you really want a top-level norms, use MultiNorms or SlowMultiReaderWrapper.
+  (Uwe Schindler, Robert Muir)
+
+* LUCENE-2837: Collapsed Searcher, Searchable into IndexSearcher;
+  removed contrib/remote and MultiSearcher (Mike McCandless); absorbed
+  ParallelMultiSearcher into IndexSearcher as an optional
+  ExecutorServiced passed to its ctor.  (Mike McCandless)
+
+* LUCENE-2831: Changed Weight#scorer, Weight#explain & Filter#getDocIdSet to
+  operate on a AtomicReaderContext instead of directly on IndexReader to enable
+  searches to be aware of IndexSearcher's context. (Simon Willnauer)
+  
+* LUCENE-2839: Scorer#score(Collector,int,int) is now public because it is
+  called from other classes and part of public API. (Uwe Schindler)
+  
+* LUCENE-2865: Weight#scorer(AtomicReaderContext, boolean, boolean) now accepts
+  a ScorerContext struct instead of booleans.(Simon Willnauer)
+
 Changes in Runtime Behavior
 
 * LUCENE-2650, LUCENE-2825: The behavior of FSDirectory.open has changed. On 64-bit
@@ -136,6 +155,17 @@ Changes in Runtime Behavior
   without any changes to the index will not cause any index version increment.
   (Simon Willnauer, Mike McCandless)
 
+* LUCENE-2846: omitNorms now behaves like omitTermFrequencyAndPositions, if you
+  omitNorms(true) for field "a" for 1000 documents, but then add a document with
+  omitNorms(false) for field "a", all documents for field "a" will have no norms.
+  Previously, Lucene would fill the first 1000 documents with "fake norms" from
+  Similarity.getDefault(). (Robert Muir, Mike Mccandless)
+
+* LUCENE-2846: When some documents contain field "a", and others do not, the
+  documents that don't have the field get a norm byte value of 0. Previously, Lucene
+  would populate "fake norms" with Similarity.getDefault() for these documents.
+  (Robert Muir, Mike Mccandless)
+
 API Changes
 
 * LUCENE-2302, LUCENE-1458, LUCENE-2111, LUCENE-2514: Terms are no longer
@@ -172,6 +202,18 @@ API Changes
 
 * LUCENE-2778: RAMDirectory now exposes newRAMFile() which allows to override
   and return a different RAMFile implementation. (Shai Erera)
+  
+* LUCENE-2831: Weight#scorer, Weight#explain, Filter#getDocIdSet,
+  Collector#setNextReader & FieldComparator#setNextReader now expect an
+  AtomicReaderContext instead of an IndexReader. (Simon Willnauer)
+
+* LUCENE-2846: Remove the deprecated IndexReader.setNorm(int, String, float).
+  This method was only syntactic sugar for setNorm(int, String, byte), but
+  using the global Similarity.getDefault().encodeNormValue.  Use the byte-based
+  method instead to ensure that the norm is encoded with your Similarity.
+  Also removed norms(String, byte[], int), which was only used by MultiReader
+  for building top-level norms. If you really need a top-level norms, use 
+  MultiNorms or SlowMultiReaderWrapper. (Robert Muir, Mike Mccandless)
 
 New features
 
@@ -311,6 +353,12 @@ New features
   extensions within the same segment. Codecs now use their per-segment codec
   ID in the file names. (Simon Willnauer)
 
+* LUCENE-2843: Added a new terms index impl,
+  VariableGapTermsIndexWriter/Reader, that accepts a pluggable
+  IndexTermSelector for picking which terms should be indexed in the
+  terms dict.  This impl stores the indexed terms in an FST, which is
+  much more RAM efficient than FixedGapTermsIndex.  (Mike McCandless)
+
 Optimizations
 
 * LUCENE-2410: ~20% speedup on exact (slop=0) PhraseQuery matching.
@@ -333,6 +381,11 @@ Optimizations
   not seek backwards when a sub-range has no terms. It now only seeks
   when the current term is less than the next sub-range's lower end.
   (Uwe Schindler, Mike McCandless)
+  
+* LUCENE-2694: Optimize MultiTermQuery to be single pass for Term lookups.
+  MultiTermQuery now stores TermState per leaf reader during rewrite to re-
+  seek the term dictionary in TermQuery / TermWeight.
+  (Simon Willnauer, Mike McCandless, Robert Muir)
 
 Documentation
 
@@ -353,7 +406,8 @@ Bug fixes
   to a mutable reference to the IndexWriters SegmentInfos. 
   (Simon Willnauer, Earwin Burrfoot)
 
-
+* LUCENE-2860: Fixed SegmentInfo.sizeInBytes to factor includeDocStores when it 
+  decides whether to return the cached computed size or not. (Shai Erera)
 
 ======================= Lucene 3.x (not yet released) =======================
 

Modified: lucene/dev/branches/realtime_search/lucene/build.xml
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/build.xml?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/build.xml (original)
+++ lucene/dev/branches/realtime_search/lucene/build.xml Thu Jan 13 19:53:21 2011
@@ -45,7 +45,7 @@
 
   <patternset id="src.dist.patterns"
               includes="src/,build.xml,*build*.xml,docs/,*.txt,contrib/,*pom.xml*,lib/,backwards/"
-              excludes="contrib/db/*/lib/,contrib/*/ext-libs/,src/site/build/,contrib/benchmark/temp/,contrib/benchmark/work/"
+              excludes="contrib/db/*/lib/,contrib/*/ext-libs/,src/site/build/"
   />
   <patternset id="binary.build.dist.patterns"
               includes="${final.name}.jar,docs/,contrib/*/*.jar,contrib/*/*.war, contrib/*/*/*.jar"
@@ -208,7 +208,6 @@
           <!-- Also remember to keep site.xml in sync.            -->		  
 
           <packageset dir="contrib/ant/src/java"/>
-          <packageset dir="contrib/benchmark/src/java"/>
           <packageset dir="contrib/db/bdb-je/src/java"/>
           <packageset dir="contrib/db/bdb/src/java"/>
           <packageset dir="contrib/demo/src/java"/>
@@ -218,7 +217,6 @@
           <packageset dir="contrib/memory/src/java"/>
           <packageset dir="contrib/misc/src/java"/>
           <packageset dir="contrib/queries/src/java"/>
-          <packageset dir="contrib/remote/src/java"/>
           <packageset dir="contrib/spatial/src/java"/>
           <packageset dir="contrib/spellchecker/src/java"/>
           <packageset dir="contrib/swing/src/java"/>
@@ -234,7 +232,6 @@
           <group title="Core" packages="org.apache.*:org.apache.lucene.analysis:org.apache.lucene.analysis.standard*:org.apache.lucene.analysis.tokenattributes*"/>
     
           <group title="contrib: Ant" packages="org.apache.lucene.ant*"/>
-          <group title="contrib: Benchmark" packages="org.apache.lucene.benchmark*"/>
           <group title="contrib: Demo" packages="org.apache.lucene.demo*"/>
           <group title="contrib: ICU" packages="org.apache.lucene.collation*"/>
           <group title="contrib: DB" packages="org.apache.lucene.store.db*:org.apache.lucene.store.je*:com.sleepycat*"/>

Modified: lucene/dev/branches/realtime_search/lucene/contrib/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/CHANGES.txt?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/CHANGES.txt (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/CHANGES.txt Thu Jan 13 19:53:21 2011
@@ -7,6 +7,8 @@ Build
  * LUCENE-2413: Moved the demo out of lucene core and into contrib/demo.
    (Robert Muir)
 
+ * LUCENE-2845: Moved contrib/benchmark to modules.
+
 New Features
 
   * LUCENE-2604: Added RegexpQuery support to contrib/queryparser.
@@ -36,6 +38,9 @@ New Features
     that doesn't synchronize on the file handle. This can be useful to 
     avoid the performance problems of SimpleFSDirectory and NIOFSDirectory.
     (Robert Muir, Simon Willnauer, Uwe Schindler, Michael McCandless)
+
+  * LUCENE-2836: Add FieldCacheRewriteMethod, which rewrites MultiTermQueries
+    using the FieldCache's TermsEnum.  (Robert Muir)
   
 API Changes
 
@@ -73,6 +78,14 @@ Changes in backwards compatibility polic
 
  * LUCENE-2581: Added new methods to FragmentsBuilder interface. These methods
    are used to set pre/post tags and Encoder. (Koji Sekiguchi)
+
+ * LUCENE-2391: Improved spellchecker (re)build time/ram usage by omitting 
+   frequencies/positions/norms for single-valued fields, modifying the default
+   ramBufferMBSize to match IndexWriterConfig (16MB), making index optimization
+   an optional boolean parameter, and modifying the incremental update logic
+   to work well with unoptimized spellcheck indexes. The indexDictionary() methods 
+   were made final to ensure a hard backwards break in case you were subclassing 
+   Spellchecker. In general, subclassing Spellchecker is not recommended.  (Robert Muir)
     
 Changes in runtime behavior
 
@@ -268,6 +281,9 @@ New features
    BooleanModifiersQueryNodeProcessor, for example instead of GroupQueryNodeProcessor.
    (Adriano Crestani via Robert Muir)
 
+ * LUCENE-2842: Add analyzer for Galician. Also adds the RSLP (Orengo) stemmer
+   for Portuguese.  (Robert Muir)
+
 Build
 
  * LUCENE-2124: Moved the JDK-based collation support from contrib/collation 

Modified: lucene/dev/branches/realtime_search/lucene/contrib/ant/src/java/org/apache/lucene/ant/IndexTask.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/ant/src/java/org/apache/lucene/ant/IndexTask.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/ant/src/java/org/apache/lucene/ant/IndexTask.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/ant/src/java/org/apache/lucene/ant/IndexTask.java Thu Jan 13 19:53:21 2011
@@ -44,7 +44,6 @@ import org.apache.lucene.index.Term;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.ScoreDoc;
-import org.apache.lucene.search.Searcher;
 import org.apache.lucene.search.TermQuery;
 import org.apache.lucene.store.FSDirectory;
 import org.apache.lucene.util.Version;
@@ -268,7 +267,7 @@ public class IndexTask extends Task {
 
     FSDirectory dir = FSDirectory.open(indexDir);
     try {
-      Searcher searcher = null;
+      IndexSearcher searcher = null;
       boolean checkLastModified = false;
       if (!create) {
         try {

Modified: lucene/dev/branches/realtime_search/lucene/contrib/ant/src/test/org/apache/lucene/ant/IndexTaskTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/ant/src/test/org/apache/lucene/ant/IndexTaskTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/ant/src/test/org/apache/lucene/ant/IndexTaskTest.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/ant/src/test/org/apache/lucene/ant/IndexTaskTest.java Thu Jan 13 19:53:21 2011
@@ -18,14 +18,13 @@ package org.apache.lucene.ant;
  */
 
 import java.io.File;
-import java.io.IOException;
+import java.io.IOException;  // javadoc
 
 import org.apache.lucene.analysis.Analyzer;
 import org.apache.lucene.analysis.core.StopAnalyzer;
 import org.apache.lucene.queryParser.QueryParser;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.Query;
-import org.apache.lucene.search.Searcher;
 import org.apache.lucene.store.Directory;
 import org.apache.tools.ant.Project;
 import org.apache.tools.ant.types.FileSet;
@@ -39,7 +38,7 @@ public class IndexTaskTest extends Lucen
     private final static String docHandler =
             "org.apache.lucene.ant.FileExtensionDocumentHandler";
 
-    private Searcher searcher;
+    private IndexSearcher searcher;
     private Analyzer analyzer;
     private Directory dir;
 

Modified: lucene/dev/branches/realtime_search/lucene/contrib/demo/src/java/org/apache/lucene/demo/SearchFiles.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/demo/src/java/org/apache/lucene/demo/SearchFiles.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/demo/src/java/org/apache/lucene/demo/SearchFiles.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/demo/src/java/org/apache/lucene/demo/SearchFiles.java Thu Jan 13 19:53:21 2011
@@ -27,15 +27,14 @@ import java.util.Date;
 import org.apache.lucene.analysis.Analyzer;
 import org.apache.lucene.analysis.standard.StandardAnalyzer;
 import org.apache.lucene.document.Document;
-import org.apache.lucene.index.FilterIndexReader;
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.queryParser.QueryParser;
 import org.apache.lucene.search.Collector;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.Query;
 import org.apache.lucene.search.ScoreDoc;
 import org.apache.lucene.search.Scorer;
-import org.apache.lucene.search.Searcher;
 import org.apache.lucene.search.TopScoreDocCollector;
 import org.apache.lucene.store.FSDirectory;
 import org.apache.lucene.util.Version;
@@ -43,31 +42,12 @@ import org.apache.lucene.util.Version;
 /** Simple command-line based search demo. */
 public class SearchFiles {
 
-  /** Use the norms from one field for all fields.  Norms are read into memory,
-   * using a byte of memory per document per searched field.  This can cause
-   * search of large collections with a large number of fields to run out of
-   * memory.  If all of the fields contain only a single token, then the norms
-   * are all identical, then single norm vector may be shared. */
-  private static class OneNormsReader extends FilterIndexReader {
-    private String field;
-
-    public OneNormsReader(IndexReader in, String field) {
-      super(in);
-      this.field = field;
-    }
-
-    @Override
-    public byte[] norms(String field) throws IOException {
-      return in.norms(this.field);
-    }
-  }
-
   private SearchFiles() {}
 
   /** Simple command-line based search demo. */
   public static void main(String[] args) throws Exception {
     String usage =
-      "Usage:\tjava org.apache.lucene.demo.SearchFiles [-index dir] [-field f] [-repeat n] [-queries file] [-raw] [-norms field] [-paging hitsPerPage]";
+      "Usage:\tjava org.apache.lucene.demo.SearchFiles [-index dir] [-field f] [-repeat n] [-queries file] [-raw] [-paging hitsPerPage]";
     usage += "\n\tSpecify 'false' for hitsPerPage to use streaming instead of paging search.";
     if (args.length > 0 && ("-h".equals(args[0]) || "-help".equals(args[0]))) {
       System.out.println(usage);
@@ -79,7 +59,6 @@ public class SearchFiles {
     String queries = null;
     int repeat = 0;
     boolean raw = false;
-    String normsField = null;
     boolean paging = true;
     int hitsPerPage = 10;
     
@@ -98,9 +77,6 @@ public class SearchFiles {
         i++;
       } else if ("-raw".equals(args[i])) {
         raw = true;
-      } else if ("-norms".equals(args[i])) {
-        normsField = args[i+1];
-        i++;
       } else if ("-paging".equals(args[i])) {
         if (args[i+1].equals("false")) {
           paging = false;
@@ -116,10 +92,7 @@ public class SearchFiles {
     
     IndexReader reader = IndexReader.open(FSDirectory.open(new File(index)), true); // only searching, so read-only=true
 
-    if (normsField != null)
-      reader = new OneNormsReader(reader, normsField);
-
-    Searcher searcher = new IndexSearcher(reader);
+    IndexSearcher searcher = new IndexSearcher(reader);
     Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
 
     BufferedReader in = null;
@@ -171,7 +144,7 @@ public class SearchFiles {
    *  This simulates the streaming search use case, where all hits are supposed to
    *  be processed, regardless of their relevance.
    */
-  public static void doStreamingSearch(final Searcher searcher, Query query) throws IOException {
+  public static void doStreamingSearch(final IndexSearcher searcher, Query query) throws IOException {
     Collector streamingHitCollector = new Collector() {
       private Scorer scorer;
       private int docBase;
@@ -188,9 +161,9 @@ public class SearchFiles {
       }
 
       @Override
-      public void setNextReader(IndexReader reader, int docBase)
+      public void setNextReader(AtomicReaderContext context)
           throws IOException {
-        this.docBase = docBase;
+        this.docBase = context.docBase;
       }
 
       @Override
@@ -213,7 +186,7 @@ public class SearchFiles {
    * is executed another time and all hits are collected.
    * 
    */
-  public static void doPagingSearch(BufferedReader in, Searcher searcher, Query query, 
+  public static void doPagingSearch(BufferedReader in, IndexSearcher searcher, Query query, 
                                      int hitsPerPage, boolean raw, boolean interactive) throws IOException {
  
     // Collect enough docs to show 5 pages

Modified: lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterPhraseTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterPhraseTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterPhraseTest.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterPhraseTest.java Thu Jan 13 19:53:21 2011
@@ -36,6 +36,7 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.index.TermPositionVector;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.search.Collector;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.PhraseQuery;
@@ -133,9 +134,9 @@ public class HighlighterPhraseTest exten
           }
 
           @Override
-          public void setNextReader(IndexReader indexreader, int i)
+          public void setNextReader(AtomicReaderContext context)
               throws IOException {
-            this.baseDoc = i;
+            this.baseDoc = context.docBase;
           }
 
           @Override

Modified: lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java Thu Jan 13 19:53:21 2011
@@ -58,7 +58,6 @@ import org.apache.lucene.search.BooleanQ
 import org.apache.lucene.search.FilteredQuery;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.MultiPhraseQuery;
-import org.apache.lucene.search.MultiSearcher;
 import org.apache.lucene.search.MultiTermQuery;
 import org.apache.lucene.search.NumericRangeQuery;
 import org.apache.lucene.search.PhraseQuery;
@@ -1301,68 +1300,6 @@ public class HighlighterTest extends Bas
     assertEquals("XHTML Encoding should have worked:", rawDocContent, decodedSnippet);
   }
 
-  public void testMultiSearcher() throws Exception {
-    // setup index 1
-    Directory ramDir1 = newDirectory();
-    IndexWriter writer1 = new IndexWriter(ramDir1, newIndexWriterConfig(
-        TEST_VERSION_CURRENT, new MockAnalyzer(MockTokenizer.SIMPLE, true, MockTokenFilter.ENGLISH_STOPSET, true)));
-    Document d = new Document();
-    Field f = new Field(FIELD_NAME, "multiOne", Field.Store.YES, Field.Index.ANALYZED);
-    d.add(f);
-    writer1.addDocument(d);
-    writer1.optimize();
-    writer1.close();
-    IndexReader reader1 = IndexReader.open(ramDir1, true);
-
-    // setup index 2
-    Directory ramDir2 = newDirectory();
-    IndexWriter writer2 = new IndexWriter(ramDir2, newIndexWriterConfig(
-        TEST_VERSION_CURRENT, new MockAnalyzer(MockTokenizer.SIMPLE, true, MockTokenFilter.ENGLISH_STOPSET, true)));
-    d = new Document();
-    f = new Field(FIELD_NAME, "multiTwo", Field.Store.YES, Field.Index.ANALYZED);
-    d.add(f);
-    writer2.addDocument(d);
-    writer2.optimize();
-    writer2.close();
-    IndexReader reader2 = IndexReader.open(ramDir2, true);
-
-    IndexSearcher searchers[] = new IndexSearcher[2];
-    searchers[0] = new IndexSearcher(ramDir1, true);
-    searchers[1] = new IndexSearcher(ramDir2, true);
-    MultiSearcher multiSearcher = new MultiSearcher(searchers);
-    QueryParser parser = new QueryParser(TEST_VERSION_CURRENT, FIELD_NAME, new MockAnalyzer(MockTokenizer.SIMPLE, true, MockTokenFilter.ENGLISH_STOPSET, true));
-    parser.setMultiTermRewriteMethod(MultiTermQuery.SCORING_BOOLEAN_QUERY_REWRITE);
-    query = parser.parse("multi*");
-    if (VERBOSE) System.out.println("Searching for: " + query.toString(FIELD_NAME));
-    // at this point the multisearcher calls combine(query[])
-    hits = multiSearcher.search(query, null, 1000);
-
-    // query = QueryParser.parse("multi*", FIELD_NAME, new StandardAnalyzer(TEST_VERSION));
-    Query expandedQueries[] = new Query[2];
-    expandedQueries[0] = query.rewrite(reader1);
-    expandedQueries[1] = query.rewrite(reader2);
-    query = query.combine(expandedQueries);
-
-    // create an instance of the highlighter with the tags used to surround
-    // highlighted text
-    Highlighter highlighter = new Highlighter(this, new QueryTermScorer(query));
-
-    for (int i = 0; i < hits.totalHits; i++) {
-      String text = multiSearcher.doc(hits.scoreDocs[i].doc).get(FIELD_NAME);
-      TokenStream tokenStream = analyzer.tokenStream(FIELD_NAME, new StringReader(text));
-      String highlightedText = highlighter.getBestFragment(tokenStream, text);
-      if (VERBOSE) System.out.println(highlightedText);
-    }
-    assertTrue("Failed to find correct number of highlights " + numHighlights + " found",
-        numHighlights == 2);
-    reader1.close();
-    reader2.close();
-    searchers[0].close();
-    searchers[1].close();
-    ramDir1.close();
-    ramDir2.close();
-  }
-
   public void testFieldSpecificHighlighting() throws Exception {
     TestHighlightRunner helper = new TestHighlightRunner() {
 

Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndex.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndex.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndex.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndex.java Thu Jan 13 19:53:21 2011
@@ -30,6 +30,7 @@ import org.apache.lucene.analysis.Analyz
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Fieldable;
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.MultiNorms;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.index.TermsEnum;
 import org.apache.lucene.index.Fields;
@@ -211,12 +212,11 @@ public class InstantiatedIndex
       }
     }
 
-
-
     // create norms
     for (String fieldName : allFieldNames) {
       if (fields == null || fields.contains(fieldName)) {
-        getNormsByFieldNameAndDocumentNumber().put(fieldName, sourceIndexReader.norms(fieldName));
+        byte norms[] = MultiNorms.norms(sourceIndexReader, fieldName);
+        getNormsByFieldNameAndDocumentNumber().put(fieldName, norms);
       }
     }
 

Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexReader.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexReader.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexReader.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexReader.java Thu Jan 13 19:53:21 2011
@@ -31,6 +31,7 @@ import java.util.Comparator;
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.FieldSelector;
 import org.apache.lucene.index.*;
+import org.apache.lucene.index.IndexReader.ReaderContext;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.BitVector;
 import org.apache.lucene.util.BytesRef;
@@ -45,6 +46,8 @@ import org.apache.lucene.util.Bits;
 public class InstantiatedIndexReader extends IndexReader {
 
   private final InstantiatedIndex index;
+  private ReaderContext context = new AtomicReaderContext(this);
+
 
   public InstantiatedIndexReader(InstantiatedIndex index) {
     super();
@@ -331,15 +334,6 @@ public class InstantiatedIndexReader ext
   }
 
   @Override
-  public void norms(String field, byte[] bytes, int offset) throws IOException {
-    byte[] norms = getIndex().getNormsByFieldNameAndDocumentNumber().get(field);
-    if (norms == null) {
-      return;
-    }
-    System.arraycopy(norms, 0, bytes, offset, norms.length);
-  }
-
-  @Override
   protected void doSetNorm(int doc, String field, byte value) throws IOException {
     if (uncommittedNormsByFieldNameAndDocumentNumber == null) {
       uncommittedNormsByFieldNameAndDocumentNumber = new HashMap<String,List<NormUpdate>>(getIndex().getNormsByFieldNameAndDocumentNumber().size());
@@ -424,6 +418,11 @@ public class InstantiatedIndexReader ext
       }
     };
   }
+  
+  @Override
+  public ReaderContext getTopReaderContext() {
+    return context;
+  }
 
   @Override
   public TermFreqVector[] getTermFreqVectors(int docNumber) throws IOException {

Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexWriter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexWriter.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexWriter.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexWriter.java Thu Jan 13 19:53:21 2011
@@ -38,6 +38,7 @@ import org.apache.lucene.analysis.Token;
 import org.apache.lucene.analysis.TokenStream;
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Fieldable;
+import org.apache.lucene.index.FieldInvertState;
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.index.TermVectorOffsetInfo;
@@ -200,9 +201,9 @@ public class InstantiatedIndexWriter imp
       byte[] oldNorms = index.getNormsByFieldNameAndDocumentNumber().get(field);
       if (oldNorms != null) {
         System.arraycopy(oldNorms, 0, norms, 0, oldNorms.length);
-        Arrays.fill(norms, oldNorms.length, norms.length, similarity.encodeNormValue(1.0f));
+        Arrays.fill(norms, oldNorms.length, norms.length, (byte) 0);
       } else {
-        Arrays.fill(norms, 0, norms.length, similarity.encodeNormValue(1.0f));
+        Arrays.fill(norms, 0, norms.length, (byte) 0);
       }
       normsByFieldNameAndDocumentNumber.put(field, norms);
       fieldNames.remove(field);
@@ -210,7 +211,7 @@ public class InstantiatedIndexWriter imp
     for (String field : fieldNames) {
       //System.out.println(field);
       byte[] norms = new byte[index.getDocumentsByNumber().length + termDocumentInformationFactoryByDocument.size()];
-      Arrays.fill(norms, 0, norms.length, similarity.encodeNormValue(1.0f));
+      Arrays.fill(norms, 0, norms.length, (byte) 0);
       normsByFieldNameAndDocumentNumber.put(field, norms);
     }
     fieldNames.clear();
@@ -235,9 +236,10 @@ public class InstantiatedIndexWriter imp
         termsInDocument += eFieldTermDocInfoFactoriesByTermText.getValue().size();
 
         if (eFieldTermDocInfoFactoriesByTermText.getKey().indexed && !eFieldTermDocInfoFactoriesByTermText.getKey().omitNorms) {
-          float norm = eFieldTermDocInfoFactoriesByTermText.getKey().boost;
-          norm *= document.getDocument().getBoost();
-          norm *= similarity.lengthNorm(eFieldTermDocInfoFactoriesByTermText.getKey().fieldName, eFieldTermDocInfoFactoriesByTermText.getKey().fieldLength);
+          final FieldInvertState invertState = new FieldInvertState();
+          invertState.setBoost(eFieldTermDocInfoFactoriesByTermText.getKey().boost * document.getDocument().getBoost());
+          invertState.setLength(eFieldTermDocInfoFactoriesByTermText.getKey().fieldLength);
+          final float norm = similarity.computeNorm(eFieldTermDocInfoFactoriesByTermText.getKey().fieldName, invertState);
           normsByFieldNameAndDocumentNumber.get(eFieldTermDocInfoFactoriesByTermText.getKey().fieldName)[document.getDocumentNumber()] = similarity.encodeNormValue(norm);
         } else {
           System.currentTimeMillis();

Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedTermsEnum.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedTermsEnum.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedTermsEnum.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedTermsEnum.java Thu Jan 13 19:53:21 2011
@@ -18,10 +18,15 @@ package org.apache.lucene.store.instanti
 
 import org.apache.lucene.util.Bits;
 import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.index.OrdTermState;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.index.TermState;
 import org.apache.lucene.index.TermsEnum;
 import org.apache.lucene.index.DocsEnum;
 import org.apache.lucene.index.DocsAndPositionsEnum;
+import org.apache.lucene.index.codecs.PrefixCodedTermState;
+
+import java.io.IOException;
 import java.util.Arrays;
 import java.util.Comparator;
 
@@ -91,10 +96,6 @@ public class InstantiatedTermsEnum exten
   }
 
   @Override
-  public void cacheCurrentTerm() {
-  }
-
-  @Override
   public BytesRef term() {
     return br;
   }
@@ -129,5 +130,18 @@ public class InstantiatedTermsEnum exten
   public Comparator<BytesRef> getComparator() {
     return BytesRef.getUTF8SortedAsUnicodeComparator();
   }
+
+  @Override
+  public TermState termState() throws IOException {
+    final OrdTermState state = new OrdTermState();
+    state.ord = upto - start;
+    return state;
+  }
+
+  @Override
+  public SeekStatus seek(BytesRef term, TermState state) throws IOException {
+    assert state != null && state instanceof OrdTermState;
+    return seek(((OrdTermState)state).ord); // just use the ord for simplicity
+  }
 }
 

Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestEmptyIndex.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestEmptyIndex.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestEmptyIndex.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestEmptyIndex.java Thu Jan 13 19:53:21 2011
@@ -22,6 +22,7 @@ import org.apache.lucene.analysis.MockAn
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.index.MultiFields;
+import org.apache.lucene.index.MultiNorms;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.TermQuery;
@@ -67,16 +68,9 @@ public class TestEmptyIndex extends Luce
   }
 
   private void testNorms(IndexReader r) throws IOException {
-    byte[] norms;
-    norms = r.norms("foo");
+    byte[] norms = MultiNorms.norms(r, "foo");
     if (norms != null) {
       assertEquals(0, norms.length);
-      norms = new byte[10];
-      Arrays.fill(norms, (byte)10);
-      r.norms("foo", norms, 10);
-      for (byte b : norms) {
-        assertEquals((byte)10, b);
-      }
     }
   }
 

Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestIndicesEquals.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestIndicesEquals.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestIndicesEquals.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestIndicesEquals.java Thu Jan 13 19:53:21 2011
@@ -30,6 +30,7 @@ import org.apache.lucene.document.Docume
 import org.apache.lucene.document.Field;
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.MultiNorms;
 import org.apache.lucene.index.Payload;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.index.DocsEnum;
@@ -90,6 +91,10 @@ public class TestIndicesEquals extends L
     // create dir data
     IndexWriter indexWriter = new IndexWriter(dir, newIndexWriterConfig(
         TEST_VERSION_CURRENT, new MockAnalyzer()));
+    indexWriter.setInfoStream(VERBOSE ? System.out : null);
+    if (VERBOSE) {
+      System.out.println("TEST: make test index");
+    }
     for (int i = 0; i < 500; i++) {
       Document document = new Document();
       assembleDocument(document, i);
@@ -319,6 +324,9 @@ public class TestIndicesEquals extends L
 
   protected void testEquals(Directory aprioriIndex, InstantiatedIndex testIndex) throws Exception {
 
+    if (VERBOSE) {
+      System.out.println("TEST: testEquals");
+    }
     testTermDocsSomeMore(aprioriIndex, testIndex);
 
     IndexReader aprioriReader = IndexReader.open(aprioriIndex, false);
@@ -341,8 +349,8 @@ public class TestIndicesEquals extends L
 
       // test norms as used by normal use
 
-      byte[] aprioriNorms = aprioriReader.norms((String) field);
-      byte[] testNorms = testReader.norms((String) field);
+      byte[] aprioriNorms = MultiNorms.norms(aprioriReader, (String) field);
+      byte[] testNorms = MultiNorms.norms(testReader, (String) field);
 
       if (aprioriNorms != null) {
         assertEquals(aprioriNorms.length, testNorms.length);
@@ -350,35 +358,6 @@ public class TestIndicesEquals extends L
         for (int i = 0; i < aprioriNorms.length; i++) {
           assertEquals("norms does not equals for field " + field + " in document " + i, aprioriNorms[i], testNorms[i]);
         }
-
-        // test norms as used by multireader
-
-        aprioriNorms = new byte[aprioriReader.maxDoc()];
-        aprioriReader.norms((String) field, aprioriNorms, 0);
-
-        testNorms = new byte[testReader.maxDoc()];
-        testReader.norms((String) field, testNorms, 0);
-
-        assertEquals(aprioriNorms.length, testNorms.length);
-
-        for (int i = 0; i < aprioriNorms.length; i++) {
-          assertEquals("norms does not equals for field " + field + " in document " + i, aprioriNorms[i], testNorms[i]);
-        }
-
-
-        // test norms as used by multireader
-
-        aprioriNorms = new byte[aprioriReader.maxDoc() + 10];
-        aprioriReader.norms((String) field, aprioriNorms, 10);
-
-        testNorms = new byte[testReader.maxDoc() + 10];
-        testReader.norms((String) field, testNorms, 10);
-
-        assertEquals(aprioriNorms.length, testNorms.length);
-        
-        for (int i = 0; i < aprioriNorms.length; i++) {
-          assertEquals("norms does not equals for field " + field + " in document " + i, aprioriNorms[i], testNorms[i]);
-        }
       }
     }
 
@@ -400,6 +379,9 @@ public class TestIndicesEquals extends L
     String aprioriField;
     while((aprioriField = aprioriFieldsEnum.next()) != null) {
       String testField = testFieldsEnum.next();
+      if (VERBOSE) {
+        System.out.println("TEST: verify field=" + testField);
+      }
       assertEquals(aprioriField, testField);
 
       TermsEnum aprioriTermEnum = aprioriFieldsEnum.terms();
@@ -408,6 +390,9 @@ public class TestIndicesEquals extends L
       BytesRef aprioriText;
       while((aprioriText = aprioriTermEnum.next()) != null) {
         assertEquals(aprioriText, testTermEnum.next());
+        if (VERBOSE) {
+          System.out.println("TEST:   verify term=" + aprioriText.utf8ToString());
+        }
 
         assertTrue(aprioriTermEnum.docFreq() == testTermEnum.docFreq());
 
@@ -433,6 +418,10 @@ public class TestIndicesEquals extends L
             assertEquals(DocsEnum.NO_MORE_DOCS, testTermDocs.nextDoc());
             break;
           }
+          if (VERBOSE) {
+            System.out.println("TEST:     verify doc=" + aprioriTermDocs.docID());
+          }
+
           assertTrue(testTermDocs.nextDoc() != DocsEnum.NO_MORE_DOCS);
 
           assertEquals(aprioriTermDocs.docID(), testTermDocs.docID());
@@ -444,12 +433,19 @@ public class TestIndicesEquals extends L
         DocsAndPositionsEnum aprioriTermPositions = aprioriTermEnum.docsAndPositions(MultiFields.getDeletedDocs(aprioriReader), null);
         DocsAndPositionsEnum testTermPositions = testTermEnum.docsAndPositions(MultiFields.getDeletedDocs(testReader), null);
 
+        if (VERBOSE) {
+          System.out.println("TEST: enum1=" + aprioriTermPositions + " enum2=" + testTermPositions);
+        }
         if (aprioriTermPositions != null) {
 
           for (int docIndex = 0; docIndex < aprioriReader.maxDoc(); docIndex++) {
             boolean hasNext = aprioriTermPositions.nextDoc() != DocsEnum.NO_MORE_DOCS;
             if (hasNext) {
               assertTrue(testTermPositions.nextDoc() != DocsEnum.NO_MORE_DOCS);
+
+              if (VERBOSE) {
+                System.out.println("TEST:     verify doc=" + aprioriTermPositions.docID());
+              }
               
               assertEquals(aprioriTermPositions.freq(), testTermPositions.freq());
 
@@ -457,6 +453,10 @@ public class TestIndicesEquals extends L
                 int aprioriPos = aprioriTermPositions.nextPosition();
                 int testPos = testTermPositions.nextPosition();
 
+                if (VERBOSE) {
+                  System.out.println("TEST:       verify pos=" + aprioriPos);
+                }
+
                 assertEquals(aprioriPos, testPos);
 
                 assertEquals(aprioriTermPositions.hasPayload(), testTermPositions.hasPayload());

Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestRealTime.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestRealTime.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestRealTime.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestRealTime.java Thu Jan 13 19:53:21 2011
@@ -20,8 +20,8 @@ import org.apache.lucene.search.TermQuer
 import org.apache.lucene.search.Scorer;
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Field;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.Term;
-import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.util.LuceneTestCase;
 
 /**
@@ -67,7 +67,7 @@ public class TestRealTime extends Lucene
     @Override
     public void setScorer(Scorer scorer) {}
     @Override
-    public void setNextReader(IndexReader reader, int docBase) {}
+    public void setNextReader(AtomicReaderContext context) {}
     @Override
     public boolean acceptsDocsOutOfOrder() { return true; }
     @Override

Modified: lucene/dev/branches/realtime_search/lucene/contrib/lucli/src/java/lucli/LuceneMethods.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/lucli/src/java/lucli/LuceneMethods.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/lucli/src/java/lucli/LuceneMethods.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/lucli/src/java/lucli/LuceneMethods.java Thu Jan 13 19:53:21 2011
@@ -41,6 +41,7 @@ import org.apache.lucene.analysis.tokena
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Fieldable;
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.index.IndexWriterConfig;
 import org.apache.lucene.index.Fields;
@@ -53,11 +54,10 @@ import org.apache.lucene.queryParser.Mul
 import org.apache.lucene.queryParser.ParseException;
 import org.apache.lucene.search.Collector;
 import org.apache.lucene.search.Explanation;
-import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.Query;
 import org.apache.lucene.search.ScoreDoc;
 import org.apache.lucene.search.Scorer;
-import org.apache.lucene.search.Searcher;
+import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.store.FSDirectory;
 import org.apache.lucene.util.Version;
 import org.apache.lucene.util.BytesRef;
@@ -73,7 +73,7 @@ class LuceneMethods {
   private List<String> fields; //Fields as a vector
   private List<String> indexedFields; //Fields as a vector
   private String fieldsArray[]; //Fields as an array
-  private Searcher searcher;
+  private IndexSearcher searcher;
   private Query query; //current query string
   private String analyzerClassFQN = null; // Analyzer class, if NULL, use default Analyzer
 
@@ -233,7 +233,7 @@ class LuceneMethods {
     }
 
     @Override
-    public void setNextReader(IndexReader reader, int docBase) {}
+    public void setNextReader(AtomicReaderContext context) {}
     @Override
     public boolean acceptsDocsOutOfOrder() {
       return true;

Modified: lucene/dev/branches/realtime_search/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java Thu Jan 13 19:53:21 2011
@@ -38,6 +38,9 @@ import org.apache.lucene.document.Docume
 import org.apache.lucene.document.FieldSelector;
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.Fields;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
+import org.apache.lucene.index.OrdTermState;
+import org.apache.lucene.index.TermState;
 import org.apache.lucene.index.Terms;
 import org.apache.lucene.index.TermsEnum;
 import org.apache.lucene.index.FieldsEnum;
@@ -48,10 +51,10 @@ import org.apache.lucene.index.TermFreqV
 import org.apache.lucene.index.TermPositionVector;
 import org.apache.lucene.index.TermVectorMapper;
 import org.apache.lucene.index.FieldInvertState;
+import org.apache.lucene.index.IndexReader.ReaderContext;
 import org.apache.lucene.search.Collector;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.Query;
-import org.apache.lucene.search.Searcher;
 import org.apache.lucene.search.Scorer;
 import org.apache.lucene.search.Similarity;
 import org.apache.lucene.store.RAMDirectory; // for javadocs
@@ -421,7 +424,7 @@ public class MemoryIndex implements Seri
     if (query == null) 
       throw new IllegalArgumentException("query must not be null");
     
-    Searcher searcher = createSearcher();
+    IndexSearcher searcher = createSearcher();
     try {
       final float[] scores = new float[1]; // inits to 0.0f (no match)
       searcher.search(query, new Collector() {
@@ -443,7 +446,7 @@ public class MemoryIndex implements Seri
         }
 
         @Override
-        public void setNextReader(IndexReader reader, int docBase) { }
+        public void setNextReader(AtomicReaderContext context) { }
       });
       float score = scores[0];
       return score;
@@ -738,7 +741,8 @@ public class MemoryIndex implements Seri
    */
   private final class MemoryIndexReader extends IndexReader {
     
-    private Searcher searcher; // needed to find searcher.getSimilarity() 
+    private IndexSearcher searcher; // needed to find searcher.getSimilarity() 
+    private final ReaderContext readerInfos = new AtomicReaderContext(this);
     
     private MemoryIndexReader() {
       super(); // avoid as much superclass baggage as possible
@@ -765,6 +769,11 @@ public class MemoryIndex implements Seri
       if (DEBUG) System.err.println("MemoryIndexReader.docFreq: " + term + ", freq:" + freq);
       return freq;
     }
+    
+    @Override
+    public ReaderContext getTopReaderContext() {
+      return readerInfos;
+    }
   
     @Override
     public Fields fields() {
@@ -877,10 +886,6 @@ public class MemoryIndex implements Seri
       }
 
       @Override
-      public void cacheCurrentTerm() {
-      }
-
-      @Override
       public long ord() {
         return termUpto;
       }
@@ -910,8 +915,21 @@ public class MemoryIndex implements Seri
       public Comparator<BytesRef> getComparator() {
         return BytesRef.getUTF8SortedAsUnicodeComparator();
       }
-    }
 
+      @Override
+      public SeekStatus seek(BytesRef term, TermState state) throws IOException {
+        assert state != null;
+        return this.seek(((OrdTermState)state).ord);
+      }
+
+      @Override
+      public TermState termState() throws IOException {
+        OrdTermState ts = new OrdTermState();
+        ts.ord = termUpto;
+        return ts;
+      }
+    }
+    
     private class MemoryDocsEnum extends DocsEnum {
       private ArrayIntList positions;
       private boolean hasNext;
@@ -1135,7 +1153,7 @@ public class MemoryIndex implements Seri
       return Similarity.getDefault();
     }
     
-    private void setSearcher(Searcher searcher) {
+    private void setSearcher(IndexSearcher searcher) {
       this.searcher = searcher;
     }
     
@@ -1168,13 +1186,6 @@ public class MemoryIndex implements Seri
     }
   
     @Override
-    public void norms(String fieldName, byte[] bytes, int offset) {
-      if (DEBUG) System.err.println("MemoryIndexReader.norms*: " + fieldName);
-      byte[] norms = norms(fieldName);
-      System.arraycopy(norms, 0, bytes, offset, norms.length);
-    }
-  
-    @Override
     protected void doSetNorm(int doc, String fieldName, byte value) {
       throw new UnsupportedOperationException();
     }

Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/index/FieldNormModifier.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/index/FieldNormModifier.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/index/FieldNormModifier.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/index/FieldNormModifier.java Thu Jan 13 19:53:21 2011
@@ -33,7 +33,7 @@ import org.apache.lucene.util.ReaderUtil
 /**
  * Given a directory and a list of fields, updates the fieldNorms in place for every document.
  * 
- * If Similarity class is specified, uses its lengthNorm method to set norms.
+ * If Similarity class is specified, uses its computeNorm method to set norms.
  * If -n command line argument is used, removed field norms, as if 
  * {@link org.apache.lucene.document.Field.Index}.NO_NORMS was used.
  *
@@ -119,6 +119,7 @@ public class FieldNormModifier {
       final List<IndexReader> subReaders = new ArrayList<IndexReader>();
       ReaderUtil.gatherSubReaders(subReaders, reader);
 
+      final FieldInvertState invertState = new FieldInvertState();
       for(IndexReader subReader : subReaders) {
         final Bits delDocs = subReader.getDeletedDocs();
 
@@ -143,9 +144,11 @@ public class FieldNormModifier {
           }
         }
 
+        invertState.setBoost(1.0f);
         for (int d = 0; d < termCounts.length; d++) {
           if (delDocs == null || !delDocs.get(d)) {
-            subReader.setNorm(d, fieldName, sim.encodeNormValue(sim.lengthNorm(fieldName, termCounts[d])));
+            invertState.setLength(termCounts[d]);
+            subReader.setNorm(d, fieldName, sim.encodeNormValue(sim.computeNorm(fieldName, invertState)));
           }
         }
       }

Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java Thu Jan 13 19:53:21 2011
@@ -146,7 +146,7 @@ public class SweetSpotSimilarity extends
     else
       numTokens = state.getLength();
 
-    return state.getBoost() * lengthNorm(fieldName, numTokens);
+    return state.getBoost() * computeLengthNorm(fieldName, numTokens);
   }
 
   /**
@@ -167,8 +167,7 @@ public class SweetSpotSimilarity extends
    *
    * @see #setLengthNormFactors
    */
-  @Override
-  public float lengthNorm(String fieldName, int numTerms) {
+  public float computeLengthNorm(String fieldName, int numTerms) {
     int l = ln_min;
     int h = ln_max;
     float s = ln_steep;

Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/index/TestFieldNormModifier.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/index/TestFieldNormModifier.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/index/TestFieldNormModifier.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/index/TestFieldNormModifier.java Thu Jan 13 19:53:21 2011
@@ -23,6 +23,7 @@ import java.util.Arrays;
 import org.apache.lucene.analysis.MockAnalyzer;
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Field;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.search.Collector;
 import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.IndexSearcher;
@@ -43,8 +44,8 @@ public class TestFieldNormModifier exten
   /** inverts the normal notion of lengthNorm */
   public static Similarity s = new DefaultSimilarity() {
     @Override
-    public float lengthNorm(String fieldName, int numTokens) {
-      return numTokens;
+    public float computeNorm(String fieldName, FieldInvertState state) {
+      return state.getBoost() * (discountOverlaps ? state.getLength() - state.getNumOverlap() : state.getLength());
     }
   };
   
@@ -84,7 +85,7 @@ public class TestFieldNormModifier exten
   public void testFieldWithNoNorm() throws Exception {
     
     IndexReader r = IndexReader.open(store, false);
-    byte[] norms = r.norms("nonorm");
+    byte[] norms = MultiNorms.norms(r, "nonorm");
     
     // sanity check, norms should all be 1
     assertTrue("Whoops we have norms?", !r.hasNorms("nonorm"));
@@ -98,7 +99,7 @@ public class TestFieldNormModifier exten
     // nothing should have changed
     r = IndexReader.open(store, false);
     
-    norms = r.norms("nonorm");
+    norms = MultiNorms.norms(r, "nonorm");
     assertTrue("Whoops we have norms?", !r.hasNorms("nonorm"));
     assertNull(norms);
 
@@ -122,8 +123,8 @@ public class TestFieldNormModifier exten
         scores[doc + docBase] = scorer.score();
       }
       @Override
-      public void setNextReader(IndexReader reader, int docBase) {
-        this.docBase = docBase;
+      public void setNextReader(AtomicReaderContext context) {
+        docBase = context.docBase;
       }
       @Override
       public void setScorer(Scorer scorer) throws IOException {
@@ -157,8 +158,8 @@ public class TestFieldNormModifier exten
         scores[doc + docBase] = scorer.score();
       }
       @Override
-      public void setNextReader(IndexReader reader, int docBase) {
-        this.docBase = docBase;
+      public void setNextReader(AtomicReaderContext context) {
+        docBase = context.docBase;
       }
       @Override
       public void setScorer(Scorer scorer) throws IOException {
@@ -183,14 +184,14 @@ public class TestFieldNormModifier exten
   public void testNormKiller() throws IOException {
 
     IndexReader r = IndexReader.open(store, false);
-    byte[] oldNorms = r.norms("untokfield");    
+    byte[] oldNorms = MultiNorms.norms(r, "untokfield");    
     r.close();
     
     FieldNormModifier fnm = new FieldNormModifier(store, s);
     fnm.reSetNorms("untokfield");
 
     r = IndexReader.open(store, false);
-    byte[] newNorms = r.norms("untokfield");
+    byte[] newNorms = MultiNorms.norms(r, "untokfield");
     r.close();
     assertFalse(Arrays.equals(oldNorms, newNorms));    
 
@@ -209,8 +210,8 @@ public class TestFieldNormModifier exten
         scores[doc + docBase] = scorer.score();
       }
       @Override
-      public void setNextReader(IndexReader reader, int docBase) {
-        this.docBase = docBase;
+      public void setNextReader(AtomicReaderContext context) {
+        docBase = context.docBase;
       }
       @Override
       public void setScorer(Scorer scorer) throws IOException {

Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java Thu Jan 13 19:53:21 2011
@@ -21,13 +21,14 @@ package org.apache.lucene.misc;
 import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.Similarity;
 import org.apache.lucene.util.LuceneTestCase;
+import org.apache.lucene.index.FieldInvertState;
 
 /**
  * Test of the SweetSpotSimilarity
  */
 public class SweetSpotSimilarityTest extends LuceneTestCase {
 
-  public void testSweetSpotLengthNorm() {
+  public void testSweetSpotComputeNorm() {
   
     SweetSpotSimilarity ss = new SweetSpotSimilarity();
     ss.setLengthNormFactors(1,1,0.5f);
@@ -37,10 +38,13 @@ public class SweetSpotSimilarityTest ext
 
 
     // base case, should degrade
-  
+    final FieldInvertState invertState = new FieldInvertState();
+    invertState.setBoost(1.0f);
     for (int i = 1; i < 1000; i++) {
+      invertState.setLength(i);
       assertEquals("base case: i="+i,
-                   d.lengthNorm("foo",i), s.lengthNorm("foo",i),
+                   d.computeNorm("foo", invertState),
+                   s.computeNorm("foo", invertState),
                    0.0f);
     }
 
@@ -49,14 +53,21 @@ public class SweetSpotSimilarityTest ext
     ss.setLengthNormFactors(3,10,0.5f);
   
     for (int i = 3; i <=10; i++) {
+      invertState.setLength(i);
       assertEquals("3,10: spot i="+i,
-                   1.0f, s.lengthNorm("foo",i),
+                   1.0f,
+                   s.computeNorm("foo", invertState),
                    0.0f);
     }
   
     for (int i = 10; i < 1000; i++) {
+      invertState.setLength(i-9);
+      final float normD = d.computeNorm("foo", invertState);
+      invertState.setLength(i);
+      final float normS = s.computeNorm("foo", invertState);
       assertEquals("3,10: 10<x : i="+i,
-                   d.lengthNorm("foo",i-9), s.lengthNorm("foo",i),
+                   normD,
+                   normS,
                    0.0f);
     }
 
@@ -68,33 +79,54 @@ public class SweetSpotSimilarityTest ext
 
   
     for (int i = 3; i <=10; i++) {
+      invertState.setLength(i);
       assertEquals("f: 3,10: spot i="+i,
-                   1.0f, s.lengthNorm("foo",i),
+                   1.0f,
+                   s.computeNorm("foo", invertState),
                    0.0f);
     }
     for (int i = 10; i < 1000; i++) {
+      invertState.setLength(i-9);
+      final float normD = d.computeNorm("foo", invertState);
+      invertState.setLength(i);
+      final float normS = s.computeNorm("foo", invertState);
       assertEquals("f: 3,10: 10<x : i="+i,
-                   d.lengthNorm("foo",i-9), s.lengthNorm("foo",i),
+                   normD,
+                   normS,
                    0.0f);
     }
     for (int i = 8; i <=13; i++) {
+      invertState.setLength(i);
       assertEquals("f: 8,13: spot i="+i,
-                   1.0f, s.lengthNorm("bar",i),
+                   1.0f,
+                   s.computeNorm("bar", invertState),
                    0.0f);
     }
     for (int i = 6; i <=9; i++) {
+      invertState.setLength(i);
       assertEquals("f: 6,9: spot i="+i,
-                   1.0f, s.lengthNorm("yak",i),
+                   1.0f,
+                   s.computeNorm("yak", invertState),
                    0.0f);
     }
     for (int i = 13; i < 1000; i++) {
+      invertState.setLength(i-12);
+      final float normD = d.computeNorm("foo", invertState);
+      invertState.setLength(i);
+      final float normS = s.computeNorm("bar", invertState);
       assertEquals("f: 8,13: 13<x : i="+i,
-                   d.lengthNorm("foo",i-12), s.lengthNorm("bar",i),
+                   normD,
+                   normS,
                    0.0f);
     }
     for (int i = 9; i < 1000; i++) {
+      invertState.setLength(i-8);
+      final float normD = d.computeNorm("foo", invertState);
+      invertState.setLength(i);
+      final float normS = s.computeNorm("yak", invertState);
       assertEquals("f: 6,9: 9<x : i="+i,
-                   d.lengthNorm("foo",i-8), s.lengthNorm("yak",i),
+                   normD,
+                   normS,
                    0.0f);
     }
 
@@ -105,9 +137,12 @@ public class SweetSpotSimilarityTest ext
     ss.setLengthNormFactors("b",5,8,0.1f, false);
 
     for (int i = 9; i < 1000; i++) {
-      assertTrue("s: i="+i+" : a="+ss.lengthNorm("a",i)+
-                 " < b="+ss.lengthNorm("b",i),
-                 ss.lengthNorm("a",i) < s.lengthNorm("b",i));
+      invertState.setLength(i);
+      final float normSS = ss.computeNorm("a", invertState);
+      final float normS = s.computeNorm("b", invertState);
+      assertTrue("s: i="+i+" : a="+normSS+
+                 " < b="+normS,
+                 normSS < normS);
     }
 
   }

Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/TestLengthNormModifier.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/TestLengthNormModifier.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/TestLengthNormModifier.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/TestLengthNormModifier.java Thu Jan 13 19:53:21 2011
@@ -22,9 +22,12 @@ import java.io.IOException;
 import org.apache.lucene.analysis.MockAnalyzer;
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Field;
+import org.apache.lucene.index.FieldInvertState;
 import org.apache.lucene.index.FieldNormModifier;
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.MultiNorms;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.search.Collector;
 import org.apache.lucene.search.DefaultSimilarity;
@@ -45,11 +48,11 @@ public class TestLengthNormModifier exte
 
     /** inverts the normal notion of lengthNorm */
     public static Similarity s = new DefaultSimilarity() {
-	    @Override
-	    public float lengthNorm(String fieldName, int numTokens) {
-		return numTokens;
-	    }
-	};
+        @Override
+        public float computeNorm(String fieldName, FieldInvertState state) {
+          return state.getBoost() * (discountOverlaps ? state.getLength() - state.getNumOverlap() : state.getLength());
+        }
+      };
     
     @Override
     public void setUp() throws Exception {
@@ -94,7 +97,7 @@ public class TestLengthNormModifier exte
     public void testFieldWithNoNorm() throws Exception {
 
 	IndexReader r = IndexReader.open(store, false);
-	byte[] norms = r.norms("nonorm");
+	byte[] norms = MultiNorms.norms(r, "nonorm");
 
 	// sanity check, norms should all be 1
 	assertTrue("Whoops we have norms?", !r.hasNorms("nonorm"));
@@ -112,7 +115,7 @@ public class TestLengthNormModifier exte
 	// nothing should have changed
 	r = IndexReader.open(store, false);
 	
-	norms = r.norms("nonorm");
+	norms = MultiNorms.norms(r, "nonorm");
 	assertTrue("Whoops we have norms?", !r.hasNorms("nonorm"));
   assertNull(norms);
 
@@ -137,8 +140,8 @@ public class TestLengthNormModifier exte
       scores[doc + docBase] = scorer.score();
     }
     @Override
-    public void setNextReader(IndexReader reader, int docBase) {
-      this.docBase = docBase;
+    public void setNextReader(AtomicReaderContext context) {
+      docBase = context.docBase;
     }
     @Override
     public void setScorer(Scorer scorer) throws IOException {
@@ -161,11 +164,11 @@ public class TestLengthNormModifier exte
 
 	// override the norms to be inverted
 	Similarity s = new DefaultSimilarity() {
-		@Override
-		public float lengthNorm(String fieldName, int numTokens) {
-		    return numTokens;
-		}
-	    };
+            @Override
+            public float computeNorm(String fieldName, FieldInvertState state) {
+              return state.getBoost() * (discountOverlaps ? state.getLength() - state.getNumOverlap() : state.getLength());
+            }
+          };
 	FieldNormModifier fnm = new FieldNormModifier(store, s);
 	fnm.reSetNorms("field");
 
@@ -179,8 +182,8 @@ public class TestLengthNormModifier exte
         scores[doc + docBase] = scorer.score();
       }
       @Override
-      public void setNextReader(IndexReader reader, int docBase) {
-        this.docBase = docBase;
+      public void setNextReader(AtomicReaderContext context) {
+        docBase = context.docBase;
       }
       @Override
       public void setScorer(Scorer scorer) throws IOException {

Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BooleanFilter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BooleanFilter.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BooleanFilter.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BooleanFilter.java Thu Jan 13 19:53:21 2011
@@ -21,6 +21,7 @@ import java.io.IOException;
 import java.util.ArrayList;
 
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.search.BooleanClause.Occur;
 import org.apache.lucene.util.OpenBitSet;
 import org.apache.lucene.util.OpenBitSetDISI;
@@ -41,10 +42,10 @@ public class BooleanFilter extends Filte
   ArrayList<Filter> notFilters = null;
   ArrayList<Filter> mustFilters = null;
   
-  private DocIdSetIterator getDISI(ArrayList<Filter> filters, int index, IndexReader reader)
+  private DocIdSetIterator getDISI(ArrayList<Filter> filters, int index, AtomicReaderContext context)
   throws IOException
   {
-    return filters.get(index).getDocIdSet(reader).iterator();
+    return filters.get(index).getDocIdSet(context).iterator();
   }
 
   /**
@@ -52,21 +53,21 @@ public class BooleanFilter extends Filte
    * of the filters that have been added.
    */
   @Override
-  public DocIdSet getDocIdSet(IndexReader reader) throws IOException
+  public DocIdSet getDocIdSet(AtomicReaderContext context) throws IOException
   {
     OpenBitSetDISI res = null;
-  
+    final IndexReader reader = context.reader;
     if (shouldFilters != null) {
       for (int i = 0; i < shouldFilters.size(); i++) {
         if (res == null) {
-          res = new OpenBitSetDISI(getDISI(shouldFilters, i, reader), reader.maxDoc());
+          res = new OpenBitSetDISI(getDISI(shouldFilters, i, context), reader.maxDoc());
         } else { 
-          DocIdSet dis = shouldFilters.get(i).getDocIdSet(reader);
+          DocIdSet dis = shouldFilters.get(i).getDocIdSet(context);
           if(dis instanceof OpenBitSet) {
             // optimized case for OpenBitSets
             res.or((OpenBitSet) dis);
           } else {
-            res.inPlaceOr(getDISI(shouldFilters, i, reader));
+            res.inPlaceOr(getDISI(shouldFilters, i, context));
           }
         }
       }
@@ -75,15 +76,15 @@ public class BooleanFilter extends Filte
     if (notFilters!=null) {
       for (int i = 0; i < notFilters.size(); i++) {
         if (res == null) {
-          res = new OpenBitSetDISI(getDISI(notFilters, i, reader), reader.maxDoc());
+          res = new OpenBitSetDISI(getDISI(notFilters, i, context), reader.maxDoc());
           res.flip(0, reader.maxDoc()); // NOTE: may set bits on deleted docs
         } else {
-          DocIdSet dis = notFilters.get(i).getDocIdSet(reader);
+          DocIdSet dis = notFilters.get(i).getDocIdSet(context);
           if(dis instanceof OpenBitSet) {
             // optimized case for OpenBitSets
             res.andNot((OpenBitSet) dis);
           } else {
-            res.inPlaceNot(getDISI(notFilters, i, reader));
+            res.inPlaceNot(getDISI(notFilters, i, context));
           }
         }
       }
@@ -92,14 +93,14 @@ public class BooleanFilter extends Filte
     if (mustFilters!=null) {
       for (int i = 0; i < mustFilters.size(); i++) {
         if (res == null) {
-          res = new OpenBitSetDISI(getDISI(mustFilters, i, reader), reader.maxDoc());
+          res = new OpenBitSetDISI(getDISI(mustFilters, i, context), reader.maxDoc());
         } else {
-          DocIdSet dis = mustFilters.get(i).getDocIdSet(reader);
+          DocIdSet dis = mustFilters.get(i).getDocIdSet(context);
           if(dis instanceof OpenBitSet) {
             // optimized case for OpenBitSets
             res.and((OpenBitSet) dis);
           } else {
-            res.inPlaceAnd(getDISI(mustFilters, i, reader));
+            res.inPlaceAnd(getDISI(mustFilters, i, context));
           }
         }
       }

Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BoostingQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BoostingQuery.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BoostingQuery.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BoostingQuery.java Thu Jan 13 19:53:21 2011
@@ -23,7 +23,7 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.search.BooleanQuery;
 import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.Query;
-import org.apache.lucene.search.Searcher;
+import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.Similarity;
 /**
  * The BoostingQuery class can be used to effectively demote results that match a given query. 
@@ -58,7 +58,7 @@ public class BoostingQuery extends Query
       BooleanQuery result = new BooleanQuery() {
 
         @Override
-        public Similarity getSimilarity(Searcher searcher) {
+        public Similarity getSimilarity(IndexSearcher searcher) {
           return new DefaultSimilarity() {
 
             @Override

Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/ChainedFilter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/ChainedFilter.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/ChainedFilter.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/ChainedFilter.java Thu Jan 13 19:53:21 2011
@@ -20,6 +20,7 @@ package org.apache.lucene.search;
 import java.io.IOException;
 
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.search.DocIdSet;
 import org.apache.lucene.search.DocIdSetIterator;
 import org.apache.lucene.search.Filter;
@@ -96,21 +97,21 @@ public class ChainedFilter extends Filte
      * {@link Filter#getDocIdSet}.
      */
     @Override
-    public DocIdSet getDocIdSet(IndexReader reader) throws IOException
+    public DocIdSet getDocIdSet(AtomicReaderContext context) throws IOException
     {
         int[] index = new int[1]; // use array as reference to modifiable int; 
         index[0] = 0;             // an object attribute would not be thread safe.
         if (logic != -1)
-            return getDocIdSet(reader, logic, index);
+            return getDocIdSet(context, logic, index);
         else if (logicArray != null)
-            return getDocIdSet(reader, logicArray, index);
+            return getDocIdSet(context, logicArray, index);
         else
-            return getDocIdSet(reader, DEFAULT, index);
+            return getDocIdSet(context, DEFAULT, index);
     }
 
-    private DocIdSetIterator getDISI(Filter filter, IndexReader reader)
+    private DocIdSetIterator getDISI(Filter filter, AtomicReaderContext context)
     throws IOException {
-        DocIdSet docIdSet = filter.getDocIdSet(reader);
+        DocIdSet docIdSet = filter.getDocIdSet(context);
         if (docIdSet == null) {
           return DocIdSet.EMPTY_DOCIDSET.iterator();
         } else {
@@ -123,9 +124,10 @@ public class ChainedFilter extends Filte
         }
     }
 
-    private OpenBitSetDISI initialResult(IndexReader reader, int logic, int[] index)
+    private OpenBitSetDISI initialResult(AtomicReaderContext context, int logic, int[] index)
     throws IOException
     {
+        IndexReader reader = context.reader;
         OpenBitSetDISI result;
         /**
          * First AND operation takes place against a completely false
@@ -133,12 +135,12 @@ public class ChainedFilter extends Filte
          */
         if (logic == AND)
         {
-            result = new OpenBitSetDISI(getDISI(chain[index[0]], reader), reader.maxDoc());
+            result = new OpenBitSetDISI(getDISI(chain[index[0]], context), reader.maxDoc());
             ++index[0];
         }
         else if (logic == ANDNOT)
         {
-            result = new OpenBitSetDISI(getDISI(chain[index[0]], reader), reader.maxDoc());
+            result = new OpenBitSetDISI(getDISI(chain[index[0]], context), reader.maxDoc());
             result.flip(0,reader.maxDoc()); // NOTE: may set bits for deleted docs.
             ++index[0];
         }
@@ -155,13 +157,13 @@ public class ChainedFilter extends Filte
      * @param logic Logical operation
      * @return DocIdSet
      */
-    private DocIdSet getDocIdSet(IndexReader reader, int logic, int[] index)
+    private DocIdSet getDocIdSet(AtomicReaderContext context, int logic, int[] index)
     throws IOException
     {
-        OpenBitSetDISI result = initialResult(reader, logic, index);
+        OpenBitSetDISI result = initialResult(context, logic, index);
         for (; index[0] < chain.length; index[0]++)
         {
-            doChain(result, logic, chain[index[0]].getDocIdSet(reader));
+            doChain(result, logic, chain[index[0]].getDocIdSet(context));
         }
         return result;
     }
@@ -172,16 +174,16 @@ public class ChainedFilter extends Filte
      * @param logic Logical operation
      * @return DocIdSet
      */
-    private DocIdSet getDocIdSet(IndexReader reader, int[] logic, int[] index)
+    private DocIdSet getDocIdSet(AtomicReaderContext info, int[] logic, int[] index)
     throws IOException
     {
         if (logic.length != chain.length)
             throw new IllegalArgumentException("Invalid number of elements in logic array");
 
-        OpenBitSetDISI result = initialResult(reader, logic[0], index);
+        OpenBitSetDISI result = initialResult(info, logic[0], index);
         for (; index[0] < chain.length; index[0]++)
         {
-            doChain(result, logic[index[0]], chain[index[0]].getDocIdSet(reader));
+            doChain(result, logic[index[0]], chain[index[0]].getDocIdSet(info));
         }
         return result;
     }

Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/DuplicateFilter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/DuplicateFilter.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/DuplicateFilter.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/DuplicateFilter.java Thu Jan 13 19:53:21 2011
@@ -19,6 +19,7 @@ import java.io.IOException;
 
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.util.BytesRef;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.Terms;
 import org.apache.lucene.index.DocsEnum;
 import org.apache.lucene.index.TermsEnum;
@@ -27,7 +28,8 @@ import org.apache.lucene.util.OpenBitSet
 import org.apache.lucene.util.Bits;
 
 public class DuplicateFilter extends Filter
-{
+{ // TODO: make duplicate filter aware of ReaderContext such that we can 
+  // filter duplicates across segments
 	
 	String fieldName;
 	
@@ -70,15 +72,15 @@ public class DuplicateFilter extends Fil
 	}
 
   @Override
-  public DocIdSet getDocIdSet(IndexReader reader) throws IOException
+  public DocIdSet getDocIdSet(AtomicReaderContext context) throws IOException
 	{
 		if(processingMode==PM_FAST_INVALIDATION)
 		{
-			return fastBits(reader);
+			return fastBits(context.reader);
 		}
 		else
 		{
-			return correctBits(reader);
+			return correctBits(context.reader);
 		}
 	}
 	
@@ -96,7 +98,7 @@ public class DuplicateFilter extends Fil
         } else {
           docs = termsEnum.docs(delDocs, docs);
           int doc = docs.nextDoc();
-          if (doc != docs.NO_MORE_DOCS) {
+          if (doc != DocsEnum.NO_MORE_DOCS) {
             if (keepMode == KM_USE_FIRST_OCCURRENCE) {
               bits.set(doc);
             } else {
@@ -104,7 +106,7 @@ public class DuplicateFilter extends Fil
               while (true) {
                 lastDoc = doc;
                 doc = docs.nextDoc();
-                if (doc == docs.NO_MORE_DOCS) {
+                if (doc == DocsEnum.NO_MORE_DOCS) {
                   break;
                 }
               }
@@ -136,7 +138,7 @@ public class DuplicateFilter extends Fil
             // unset potential duplicates
             docs = termsEnum.docs(delDocs, docs);
             int doc = docs.nextDoc();
-            if (doc != docs.NO_MORE_DOCS) {
+            if (doc != DocsEnum.NO_MORE_DOCS) {
               if (keepMode == KM_USE_FIRST_OCCURRENCE) {
                 doc = docs.nextDoc();
               }
@@ -147,7 +149,7 @@ public class DuplicateFilter extends Fil
               lastDoc = doc;
               bits.clear(lastDoc);
               doc = docs.nextDoc();
-              if (doc == docs.NO_MORE_DOCS) {
+              if (doc == DocsEnum.NO_MORE_DOCS) {
                 break;
               }
             }

Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/FuzzyLikeThisQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/FuzzyLikeThisQuery.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/FuzzyLikeThisQuery.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/FuzzyLikeThisQuery.java Thu Jan 13 19:53:21 2011
@@ -292,7 +292,7 @@ public class FuzzyLikeThisQuery extends 
             {
                 //optimize where only one selected variant
                 ScoreTerm st= variants.get(0);
-                TermQuery tq = new FuzzyTermQuery(st.term,ignoreTF);
+                Query tq = ignoreTF ? new ConstantScoreQuery(new TermQuery(st.term)) : new TermQuery(st.term, 1);
                 tq.setBoost(st.score); // set the boost to a mix of IDF and score
                 bq.add(tq, BooleanClause.Occur.SHOULD); 
             }
@@ -303,7 +303,8 @@ public class FuzzyLikeThisQuery extends 
                         .hasNext();)
                 {
                     ScoreTerm st = iterator2.next();
-                    TermQuery tq = new FuzzyTermQuery(st.term,ignoreTF);      // found a match
+                    // found a match
+                    Query tq = ignoreTF ? new ConstantScoreQuery(new TermQuery(st.term)) : new TermQuery(st.term, 1);                    
                     tq.setBoost(st.score); // set the boost using the ScoreTerm's score
                     termVariants.add(tq, BooleanClause.Occur.SHOULD);          // add to query                    
                 }
@@ -348,45 +349,8 @@ public class FuzzyLikeThisQuery extends 
             return termA.score < termB.score;
         }
         
-      }
-      
-      //overrides basic TermQuery to negate effects of IDF (idf is factored into boost of containing BooleanQuery)
-      private static class FuzzyTermQuery extends TermQuery
-      {
-    	  boolean ignoreTF;
-          public FuzzyTermQuery(Term t, boolean ignoreTF)
-          {
-        	  super(t);
-        	  this.ignoreTF=ignoreTF;
-          }
-          @Override
-          public Similarity getSimilarity(Searcher searcher)
-          {            
-              Similarity result = super.getSimilarity(searcher);
-              result = new SimilarityDelegator(result) {
-                  
-                  @Override
-                  public float tf(float freq)
-                  {
-                	  if(ignoreTF)
-                	  {
-                          return 1; //ignore tf
-                	  }
-            		  return super.tf(freq);
-                  }
-                  @Override
-                  public float idf(int docFreq, int numDocs)
-                  {
-                      //IDF is already factored into individual term boosts
-                      return 1;
-                  }               
-              };
-              return result;
-          }        
-      }
+      }    
       
-      
-
     /* (non-Javadoc)
      * @see org.apache.lucene.search.Query#toString(java.lang.String)
      */

Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/TermsFilter.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/TermsFilter.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/TermsFilter.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/TermsFilter.java Thu Jan 13 19:53:21 2011
@@ -23,6 +23,7 @@ import java.util.Set;
 import java.util.TreeSet;
 
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.index.DocsEnum;
 import org.apache.lucene.index.Terms;
@@ -57,7 +58,8 @@ public class TermsFilter extends Filter
    * @see org.apache.lucene.search.Filter#getDocIdSet(org.apache.lucene.index.IndexReader)
 	 */
   @Override
-  public DocIdSet getDocIdSet(IndexReader reader) throws IOException {
+  public DocIdSet getDocIdSet(AtomicReaderContext context) throws IOException {
+    IndexReader reader = context.reader;
     OpenBitSet result=new OpenBitSet(reader.maxDoc());
     Fields fields = reader.fields();
     BytesRef br = new BytesRef();

Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/BooleanFilterTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/BooleanFilterTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/BooleanFilterTest.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/BooleanFilterTest.java Thu Jan 13 19:53:21 2011
@@ -24,6 +24,7 @@ import org.apache.lucene.analysis.MockTo
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Field;
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.SlowMultiReaderWrapper;
 import org.apache.lucene.index.Term;
@@ -83,7 +84,7 @@ public class BooleanFilterTest extends L
         private void tstFilterCard(String mes, int expected, Filter filt)
         throws Throwable
         {
-          DocIdSetIterator disi = filt.getDocIdSet(reader).iterator();
+          DocIdSetIterator disi = filt.getDocIdSet(new AtomicReaderContext(reader)).iterator();
           int actual = 0;
           while (disi.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) {
             actual++;

Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/ChainedFilterTest.java
URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/ChainedFilterTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff
==============================================================================
--- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/ChainedFilterTest.java (original)
+++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/ChainedFilterTest.java Thu Jan 13 19:53:21 2011
@@ -29,11 +29,10 @@ import org.apache.lucene.search.BooleanC
 import org.apache.lucene.search.BooleanQuery;
 import org.apache.lucene.search.CachingWrapperFilter;
 import org.apache.lucene.search.Filter;
-import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.MatchAllDocsQuery;
 import org.apache.lucene.search.Query;
 import org.apache.lucene.search.QueryWrapperFilter;
-import org.apache.lucene.search.Searcher;
+import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.TermQuery;
 import org.apache.lucene.search.TermRangeFilter;
 import org.apache.lucene.search.TopDocs;
@@ -195,7 +194,7 @@ public class ChainedFilterTest extends L
     IndexReader reader = writer.getReader();
     writer.close();
   
-    Searcher searcher = new IndexSearcher(reader);
+    IndexSearcher searcher = new IndexSearcher(reader);
   
     Query query = new TermQuery(new Term("none", "none"));
   



Mime
View raw message