Return-Path: Delivered-To: apmail-lucene-commits-archive@www.apache.org Received: (qmail 68135 invoked from network); 13 Jan 2011 19:54:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 13 Jan 2011 19:54:26 -0000 Received: (qmail 96662 invoked by uid 500); 13 Jan 2011 19:54:26 -0000 Mailing-List: contact commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list commits@lucene.apache.org Received: (qmail 96655 invoked by uid 99); 13 Jan 2011 19:54:26 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 19:54:26 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 13 Jan 2011 19:54:15 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 0696723889E9; Thu, 13 Jan 2011 19:53:47 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1058718 [2/18] - in /lucene/dev/branches/realtime_search: ./ lucene/ lucene/contrib/ lucene/contrib/ant/src/java/org/apache/lucene/ant/ lucene/contrib/ant/src/test/org/apache/lucene/ant/ lucene/contrib/benchmark/ lucene/contrib/demo/src/ja... Date: Thu, 13 Jan 2011 19:53:39 -0000 To: commits@lucene.apache.org From: buschmi@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20110113195347.0696723889E9@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Modified: lucene/dev/branches/realtime_search/lucene/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/CHANGES.txt?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/CHANGES.txt (original) +++ lucene/dev/branches/realtime_search/lucene/CHANGES.txt Thu Jan 13 19:53:21 2011 @@ -119,6 +119,25 @@ Changes in backwards compatibility polic need to change it (e.g. using "\\" to escape '\' itself). (Sunil Kamath, Terry Yang via Robert Muir) +* LUCENE-2771: IndexReader.norms() now throws UOE on non-atomic IndexReaders. If + you really want a top-level norms, use MultiNorms or SlowMultiReaderWrapper. + (Uwe Schindler, Robert Muir) + +* LUCENE-2837: Collapsed Searcher, Searchable into IndexSearcher; + removed contrib/remote and MultiSearcher (Mike McCandless); absorbed + ParallelMultiSearcher into IndexSearcher as an optional + ExecutorServiced passed to its ctor. (Mike McCandless) + +* LUCENE-2831: Changed Weight#scorer, Weight#explain & Filter#getDocIdSet to + operate on a AtomicReaderContext instead of directly on IndexReader to enable + searches to be aware of IndexSearcher's context. (Simon Willnauer) + +* LUCENE-2839: Scorer#score(Collector,int,int) is now public because it is + called from other classes and part of public API. (Uwe Schindler) + +* LUCENE-2865: Weight#scorer(AtomicReaderContext, boolean, boolean) now accepts + a ScorerContext struct instead of booleans.(Simon Willnauer) + Changes in Runtime Behavior * LUCENE-2650, LUCENE-2825: The behavior of FSDirectory.open has changed. On 64-bit @@ -136,6 +155,17 @@ Changes in Runtime Behavior without any changes to the index will not cause any index version increment. (Simon Willnauer, Mike McCandless) +* LUCENE-2846: omitNorms now behaves like omitTermFrequencyAndPositions, if you + omitNorms(true) for field "a" for 1000 documents, but then add a document with + omitNorms(false) for field "a", all documents for field "a" will have no norms. + Previously, Lucene would fill the first 1000 documents with "fake norms" from + Similarity.getDefault(). (Robert Muir, Mike Mccandless) + +* LUCENE-2846: When some documents contain field "a", and others do not, the + documents that don't have the field get a norm byte value of 0. Previously, Lucene + would populate "fake norms" with Similarity.getDefault() for these documents. + (Robert Muir, Mike Mccandless) + API Changes * LUCENE-2302, LUCENE-1458, LUCENE-2111, LUCENE-2514: Terms are no longer @@ -172,6 +202,18 @@ API Changes * LUCENE-2778: RAMDirectory now exposes newRAMFile() which allows to override and return a different RAMFile implementation. (Shai Erera) + +* LUCENE-2831: Weight#scorer, Weight#explain, Filter#getDocIdSet, + Collector#setNextReader & FieldComparator#setNextReader now expect an + AtomicReaderContext instead of an IndexReader. (Simon Willnauer) + +* LUCENE-2846: Remove the deprecated IndexReader.setNorm(int, String, float). + This method was only syntactic sugar for setNorm(int, String, byte), but + using the global Similarity.getDefault().encodeNormValue. Use the byte-based + method instead to ensure that the norm is encoded with your Similarity. + Also removed norms(String, byte[], int), which was only used by MultiReader + for building top-level norms. If you really need a top-level norms, use + MultiNorms or SlowMultiReaderWrapper. (Robert Muir, Mike Mccandless) New features @@ -311,6 +353,12 @@ New features extensions within the same segment. Codecs now use their per-segment codec ID in the file names. (Simon Willnauer) +* LUCENE-2843: Added a new terms index impl, + VariableGapTermsIndexWriter/Reader, that accepts a pluggable + IndexTermSelector for picking which terms should be indexed in the + terms dict. This impl stores the indexed terms in an FST, which is + much more RAM efficient than FixedGapTermsIndex. (Mike McCandless) + Optimizations * LUCENE-2410: ~20% speedup on exact (slop=0) PhraseQuery matching. @@ -333,6 +381,11 @@ Optimizations not seek backwards when a sub-range has no terms. It now only seeks when the current term is less than the next sub-range's lower end. (Uwe Schindler, Mike McCandless) + +* LUCENE-2694: Optimize MultiTermQuery to be single pass for Term lookups. + MultiTermQuery now stores TermState per leaf reader during rewrite to re- + seek the term dictionary in TermQuery / TermWeight. + (Simon Willnauer, Mike McCandless, Robert Muir) Documentation @@ -353,7 +406,8 @@ Bug fixes to a mutable reference to the IndexWriters SegmentInfos. (Simon Willnauer, Earwin Burrfoot) - +* LUCENE-2860: Fixed SegmentInfo.sizeInBytes to factor includeDocStores when it + decides whether to return the cached computed size or not. (Shai Erera) ======================= Lucene 3.x (not yet released) ======================= Modified: lucene/dev/branches/realtime_search/lucene/build.xml URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/build.xml?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/build.xml (original) +++ lucene/dev/branches/realtime_search/lucene/build.xml Thu Jan 13 19:53:21 2011 @@ -45,7 +45,7 @@ - @@ -218,7 +217,6 @@ - @@ -234,7 +232,6 @@ - Modified: lucene/dev/branches/realtime_search/lucene/contrib/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/CHANGES.txt?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/CHANGES.txt (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/CHANGES.txt Thu Jan 13 19:53:21 2011 @@ -7,6 +7,8 @@ Build * LUCENE-2413: Moved the demo out of lucene core and into contrib/demo. (Robert Muir) + * LUCENE-2845: Moved contrib/benchmark to modules. + New Features * LUCENE-2604: Added RegexpQuery support to contrib/queryparser. @@ -36,6 +38,9 @@ New Features that doesn't synchronize on the file handle. This can be useful to avoid the performance problems of SimpleFSDirectory and NIOFSDirectory. (Robert Muir, Simon Willnauer, Uwe Schindler, Michael McCandless) + + * LUCENE-2836: Add FieldCacheRewriteMethod, which rewrites MultiTermQueries + using the FieldCache's TermsEnum. (Robert Muir) API Changes @@ -73,6 +78,14 @@ Changes in backwards compatibility polic * LUCENE-2581: Added new methods to FragmentsBuilder interface. These methods are used to set pre/post tags and Encoder. (Koji Sekiguchi) + + * LUCENE-2391: Improved spellchecker (re)build time/ram usage by omitting + frequencies/positions/norms for single-valued fields, modifying the default + ramBufferMBSize to match IndexWriterConfig (16MB), making index optimization + an optional boolean parameter, and modifying the incremental update logic + to work well with unoptimized spellcheck indexes. The indexDictionary() methods + were made final to ensure a hard backwards break in case you were subclassing + Spellchecker. In general, subclassing Spellchecker is not recommended. (Robert Muir) Changes in runtime behavior @@ -268,6 +281,9 @@ New features BooleanModifiersQueryNodeProcessor, for example instead of GroupQueryNodeProcessor. (Adriano Crestani via Robert Muir) + * LUCENE-2842: Add analyzer for Galician. Also adds the RSLP (Orengo) stemmer + for Portuguese. (Robert Muir) + Build * LUCENE-2124: Moved the JDK-based collation support from contrib/collation Modified: lucene/dev/branches/realtime_search/lucene/contrib/ant/src/java/org/apache/lucene/ant/IndexTask.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/ant/src/java/org/apache/lucene/ant/IndexTask.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/ant/src/java/org/apache/lucene/ant/IndexTask.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/ant/src/java/org/apache/lucene/ant/IndexTask.java Thu Jan 13 19:53:21 2011 @@ -44,7 +44,6 @@ import org.apache.lucene.index.Term; import org.apache.lucene.index.IndexWriterConfig.OpenMode; import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.ScoreDoc; -import org.apache.lucene.search.Searcher; import org.apache.lucene.search.TermQuery; import org.apache.lucene.store.FSDirectory; import org.apache.lucene.util.Version; @@ -268,7 +267,7 @@ public class IndexTask extends Task { FSDirectory dir = FSDirectory.open(indexDir); try { - Searcher searcher = null; + IndexSearcher searcher = null; boolean checkLastModified = false; if (!create) { try { Modified: lucene/dev/branches/realtime_search/lucene/contrib/ant/src/test/org/apache/lucene/ant/IndexTaskTest.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/ant/src/test/org/apache/lucene/ant/IndexTaskTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/ant/src/test/org/apache/lucene/ant/IndexTaskTest.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/ant/src/test/org/apache/lucene/ant/IndexTaskTest.java Thu Jan 13 19:53:21 2011 @@ -18,14 +18,13 @@ package org.apache.lucene.ant; */ import java.io.File; -import java.io.IOException; +import java.io.IOException; // javadoc import org.apache.lucene.analysis.Analyzer; import org.apache.lucene.analysis.core.StopAnalyzer; import org.apache.lucene.queryParser.QueryParser; import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.Query; -import org.apache.lucene.search.Searcher; import org.apache.lucene.store.Directory; import org.apache.tools.ant.Project; import org.apache.tools.ant.types.FileSet; @@ -39,7 +38,7 @@ public class IndexTaskTest extends Lucen private final static String docHandler = "org.apache.lucene.ant.FileExtensionDocumentHandler"; - private Searcher searcher; + private IndexSearcher searcher; private Analyzer analyzer; private Directory dir; Modified: lucene/dev/branches/realtime_search/lucene/contrib/demo/src/java/org/apache/lucene/demo/SearchFiles.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/demo/src/java/org/apache/lucene/demo/SearchFiles.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/demo/src/java/org/apache/lucene/demo/SearchFiles.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/demo/src/java/org/apache/lucene/demo/SearchFiles.java Thu Jan 13 19:53:21 2011 @@ -27,15 +27,14 @@ import java.util.Date; import org.apache.lucene.analysis.Analyzer; import org.apache.lucene.analysis.standard.StandardAnalyzer; import org.apache.lucene.document.Document; -import org.apache.lucene.index.FilterIndexReader; import org.apache.lucene.index.IndexReader; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.queryParser.QueryParser; import org.apache.lucene.search.Collector; import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.Query; import org.apache.lucene.search.ScoreDoc; import org.apache.lucene.search.Scorer; -import org.apache.lucene.search.Searcher; import org.apache.lucene.search.TopScoreDocCollector; import org.apache.lucene.store.FSDirectory; import org.apache.lucene.util.Version; @@ -43,31 +42,12 @@ import org.apache.lucene.util.Version; /** Simple command-line based search demo. */ public class SearchFiles { - /** Use the norms from one field for all fields. Norms are read into memory, - * using a byte of memory per document per searched field. This can cause - * search of large collections with a large number of fields to run out of - * memory. If all of the fields contain only a single token, then the norms - * are all identical, then single norm vector may be shared. */ - private static class OneNormsReader extends FilterIndexReader { - private String field; - - public OneNormsReader(IndexReader in, String field) { - super(in); - this.field = field; - } - - @Override - public byte[] norms(String field) throws IOException { - return in.norms(this.field); - } - } - private SearchFiles() {} /** Simple command-line based search demo. */ public static void main(String[] args) throws Exception { String usage = - "Usage:\tjava org.apache.lucene.demo.SearchFiles [-index dir] [-field f] [-repeat n] [-queries file] [-raw] [-norms field] [-paging hitsPerPage]"; + "Usage:\tjava org.apache.lucene.demo.SearchFiles [-index dir] [-field f] [-repeat n] [-queries file] [-raw] [-paging hitsPerPage]"; usage += "\n\tSpecify 'false' for hitsPerPage to use streaming instead of paging search."; if (args.length > 0 && ("-h".equals(args[0]) || "-help".equals(args[0]))) { System.out.println(usage); @@ -79,7 +59,6 @@ public class SearchFiles { String queries = null; int repeat = 0; boolean raw = false; - String normsField = null; boolean paging = true; int hitsPerPage = 10; @@ -98,9 +77,6 @@ public class SearchFiles { i++; } else if ("-raw".equals(args[i])) { raw = true; - } else if ("-norms".equals(args[i])) { - normsField = args[i+1]; - i++; } else if ("-paging".equals(args[i])) { if (args[i+1].equals("false")) { paging = false; @@ -116,10 +92,7 @@ public class SearchFiles { IndexReader reader = IndexReader.open(FSDirectory.open(new File(index)), true); // only searching, so read-only=true - if (normsField != null) - reader = new OneNormsReader(reader, normsField); - - Searcher searcher = new IndexSearcher(reader); + IndexSearcher searcher = new IndexSearcher(reader); Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT); BufferedReader in = null; @@ -171,7 +144,7 @@ public class SearchFiles { * This simulates the streaming search use case, where all hits are supposed to * be processed, regardless of their relevance. */ - public static void doStreamingSearch(final Searcher searcher, Query query) throws IOException { + public static void doStreamingSearch(final IndexSearcher searcher, Query query) throws IOException { Collector streamingHitCollector = new Collector() { private Scorer scorer; private int docBase; @@ -188,9 +161,9 @@ public class SearchFiles { } @Override - public void setNextReader(IndexReader reader, int docBase) + public void setNextReader(AtomicReaderContext context) throws IOException { - this.docBase = docBase; + this.docBase = context.docBase; } @Override @@ -213,7 +186,7 @@ public class SearchFiles { * is executed another time and all hits are collected. * */ - public static void doPagingSearch(BufferedReader in, Searcher searcher, Query query, + public static void doPagingSearch(BufferedReader in, IndexSearcher searcher, Query query, int hitsPerPage, boolean raw, boolean interactive) throws IOException { // Collect enough docs to show 5 pages Modified: lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterPhraseTest.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterPhraseTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterPhraseTest.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterPhraseTest.java Thu Jan 13 19:53:21 2011 @@ -36,6 +36,7 @@ import org.apache.lucene.index.IndexRead import org.apache.lucene.index.IndexWriter; import org.apache.lucene.index.Term; import org.apache.lucene.index.TermPositionVector; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.search.Collector; import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.PhraseQuery; @@ -133,9 +134,9 @@ public class HighlighterPhraseTest exten } @Override - public void setNextReader(IndexReader indexreader, int i) + public void setNextReader(AtomicReaderContext context) throws IOException { - this.baseDoc = i; + this.baseDoc = context.docBase; } @Override Modified: lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/highlighter/src/test/org/apache/lucene/search/highlight/HighlighterTest.java Thu Jan 13 19:53:21 2011 @@ -58,7 +58,6 @@ import org.apache.lucene.search.BooleanQ import org.apache.lucene.search.FilteredQuery; import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.MultiPhraseQuery; -import org.apache.lucene.search.MultiSearcher; import org.apache.lucene.search.MultiTermQuery; import org.apache.lucene.search.NumericRangeQuery; import org.apache.lucene.search.PhraseQuery; @@ -1301,68 +1300,6 @@ public class HighlighterTest extends Bas assertEquals("XHTML Encoding should have worked:", rawDocContent, decodedSnippet); } - public void testMultiSearcher() throws Exception { - // setup index 1 - Directory ramDir1 = newDirectory(); - IndexWriter writer1 = new IndexWriter(ramDir1, newIndexWriterConfig( - TEST_VERSION_CURRENT, new MockAnalyzer(MockTokenizer.SIMPLE, true, MockTokenFilter.ENGLISH_STOPSET, true))); - Document d = new Document(); - Field f = new Field(FIELD_NAME, "multiOne", Field.Store.YES, Field.Index.ANALYZED); - d.add(f); - writer1.addDocument(d); - writer1.optimize(); - writer1.close(); - IndexReader reader1 = IndexReader.open(ramDir1, true); - - // setup index 2 - Directory ramDir2 = newDirectory(); - IndexWriter writer2 = new IndexWriter(ramDir2, newIndexWriterConfig( - TEST_VERSION_CURRENT, new MockAnalyzer(MockTokenizer.SIMPLE, true, MockTokenFilter.ENGLISH_STOPSET, true))); - d = new Document(); - f = new Field(FIELD_NAME, "multiTwo", Field.Store.YES, Field.Index.ANALYZED); - d.add(f); - writer2.addDocument(d); - writer2.optimize(); - writer2.close(); - IndexReader reader2 = IndexReader.open(ramDir2, true); - - IndexSearcher searchers[] = new IndexSearcher[2]; - searchers[0] = new IndexSearcher(ramDir1, true); - searchers[1] = new IndexSearcher(ramDir2, true); - MultiSearcher multiSearcher = new MultiSearcher(searchers); - QueryParser parser = new QueryParser(TEST_VERSION_CURRENT, FIELD_NAME, new MockAnalyzer(MockTokenizer.SIMPLE, true, MockTokenFilter.ENGLISH_STOPSET, true)); - parser.setMultiTermRewriteMethod(MultiTermQuery.SCORING_BOOLEAN_QUERY_REWRITE); - query = parser.parse("multi*"); - if (VERBOSE) System.out.println("Searching for: " + query.toString(FIELD_NAME)); - // at this point the multisearcher calls combine(query[]) - hits = multiSearcher.search(query, null, 1000); - - // query = QueryParser.parse("multi*", FIELD_NAME, new StandardAnalyzer(TEST_VERSION)); - Query expandedQueries[] = new Query[2]; - expandedQueries[0] = query.rewrite(reader1); - expandedQueries[1] = query.rewrite(reader2); - query = query.combine(expandedQueries); - - // create an instance of the highlighter with the tags used to surround - // highlighted text - Highlighter highlighter = new Highlighter(this, new QueryTermScorer(query)); - - for (int i = 0; i < hits.totalHits; i++) { - String text = multiSearcher.doc(hits.scoreDocs[i].doc).get(FIELD_NAME); - TokenStream tokenStream = analyzer.tokenStream(FIELD_NAME, new StringReader(text)); - String highlightedText = highlighter.getBestFragment(tokenStream, text); - if (VERBOSE) System.out.println(highlightedText); - } - assertTrue("Failed to find correct number of highlights " + numHighlights + " found", - numHighlights == 2); - reader1.close(); - reader2.close(); - searchers[0].close(); - searchers[1].close(); - ramDir1.close(); - ramDir2.close(); - } - public void testFieldSpecificHighlighting() throws Exception { TestHighlightRunner helper = new TestHighlightRunner() { Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndex.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndex.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndex.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndex.java Thu Jan 13 19:53:21 2011 @@ -30,6 +30,7 @@ import org.apache.lucene.analysis.Analyz import org.apache.lucene.document.Document; import org.apache.lucene.document.Fieldable; import org.apache.lucene.index.IndexReader; +import org.apache.lucene.index.MultiNorms; import org.apache.lucene.index.Term; import org.apache.lucene.index.TermsEnum; import org.apache.lucene.index.Fields; @@ -211,12 +212,11 @@ public class InstantiatedIndex } } - - // create norms for (String fieldName : allFieldNames) { if (fields == null || fields.contains(fieldName)) { - getNormsByFieldNameAndDocumentNumber().put(fieldName, sourceIndexReader.norms(fieldName)); + byte norms[] = MultiNorms.norms(sourceIndexReader, fieldName); + getNormsByFieldNameAndDocumentNumber().put(fieldName, norms); } } Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexReader.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexReader.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexReader.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexReader.java Thu Jan 13 19:53:21 2011 @@ -31,6 +31,7 @@ import java.util.Comparator; import org.apache.lucene.document.Document; import org.apache.lucene.document.FieldSelector; import org.apache.lucene.index.*; +import org.apache.lucene.index.IndexReader.ReaderContext; import org.apache.lucene.store.Directory; import org.apache.lucene.util.BitVector; import org.apache.lucene.util.BytesRef; @@ -45,6 +46,8 @@ import org.apache.lucene.util.Bits; public class InstantiatedIndexReader extends IndexReader { private final InstantiatedIndex index; + private ReaderContext context = new AtomicReaderContext(this); + public InstantiatedIndexReader(InstantiatedIndex index) { super(); @@ -331,15 +334,6 @@ public class InstantiatedIndexReader ext } @Override - public void norms(String field, byte[] bytes, int offset) throws IOException { - byte[] norms = getIndex().getNormsByFieldNameAndDocumentNumber().get(field); - if (norms == null) { - return; - } - System.arraycopy(norms, 0, bytes, offset, norms.length); - } - - @Override protected void doSetNorm(int doc, String field, byte value) throws IOException { if (uncommittedNormsByFieldNameAndDocumentNumber == null) { uncommittedNormsByFieldNameAndDocumentNumber = new HashMap>(getIndex().getNormsByFieldNameAndDocumentNumber().size()); @@ -424,6 +418,11 @@ public class InstantiatedIndexReader ext } }; } + + @Override + public ReaderContext getTopReaderContext() { + return context; + } @Override public TermFreqVector[] getTermFreqVectors(int docNumber) throws IOException { Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexWriter.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexWriter.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexWriter.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedIndexWriter.java Thu Jan 13 19:53:21 2011 @@ -38,6 +38,7 @@ import org.apache.lucene.analysis.Token; import org.apache.lucene.analysis.TokenStream; import org.apache.lucene.document.Document; import org.apache.lucene.document.Fieldable; +import org.apache.lucene.index.FieldInvertState; import org.apache.lucene.index.IndexReader; import org.apache.lucene.index.Term; import org.apache.lucene.index.TermVectorOffsetInfo; @@ -200,9 +201,9 @@ public class InstantiatedIndexWriter imp byte[] oldNorms = index.getNormsByFieldNameAndDocumentNumber().get(field); if (oldNorms != null) { System.arraycopy(oldNorms, 0, norms, 0, oldNorms.length); - Arrays.fill(norms, oldNorms.length, norms.length, similarity.encodeNormValue(1.0f)); + Arrays.fill(norms, oldNorms.length, norms.length, (byte) 0); } else { - Arrays.fill(norms, 0, norms.length, similarity.encodeNormValue(1.0f)); + Arrays.fill(norms, 0, norms.length, (byte) 0); } normsByFieldNameAndDocumentNumber.put(field, norms); fieldNames.remove(field); @@ -210,7 +211,7 @@ public class InstantiatedIndexWriter imp for (String field : fieldNames) { //System.out.println(field); byte[] norms = new byte[index.getDocumentsByNumber().length + termDocumentInformationFactoryByDocument.size()]; - Arrays.fill(norms, 0, norms.length, similarity.encodeNormValue(1.0f)); + Arrays.fill(norms, 0, norms.length, (byte) 0); normsByFieldNameAndDocumentNumber.put(field, norms); } fieldNames.clear(); @@ -235,9 +236,10 @@ public class InstantiatedIndexWriter imp termsInDocument += eFieldTermDocInfoFactoriesByTermText.getValue().size(); if (eFieldTermDocInfoFactoriesByTermText.getKey().indexed && !eFieldTermDocInfoFactoriesByTermText.getKey().omitNorms) { - float norm = eFieldTermDocInfoFactoriesByTermText.getKey().boost; - norm *= document.getDocument().getBoost(); - norm *= similarity.lengthNorm(eFieldTermDocInfoFactoriesByTermText.getKey().fieldName, eFieldTermDocInfoFactoriesByTermText.getKey().fieldLength); + final FieldInvertState invertState = new FieldInvertState(); + invertState.setBoost(eFieldTermDocInfoFactoriesByTermText.getKey().boost * document.getDocument().getBoost()); + invertState.setLength(eFieldTermDocInfoFactoriesByTermText.getKey().fieldLength); + final float norm = similarity.computeNorm(eFieldTermDocInfoFactoriesByTermText.getKey().fieldName, invertState); normsByFieldNameAndDocumentNumber.get(eFieldTermDocInfoFactoriesByTermText.getKey().fieldName)[document.getDocumentNumber()] = similarity.encodeNormValue(norm); } else { System.currentTimeMillis(); Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedTermsEnum.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedTermsEnum.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedTermsEnum.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/InstantiatedTermsEnum.java Thu Jan 13 19:53:21 2011 @@ -18,10 +18,15 @@ package org.apache.lucene.store.instanti import org.apache.lucene.util.Bits; import org.apache.lucene.util.BytesRef; +import org.apache.lucene.index.OrdTermState; import org.apache.lucene.index.Term; +import org.apache.lucene.index.TermState; import org.apache.lucene.index.TermsEnum; import org.apache.lucene.index.DocsEnum; import org.apache.lucene.index.DocsAndPositionsEnum; +import org.apache.lucene.index.codecs.PrefixCodedTermState; + +import java.io.IOException; import java.util.Arrays; import java.util.Comparator; @@ -91,10 +96,6 @@ public class InstantiatedTermsEnum exten } @Override - public void cacheCurrentTerm() { - } - - @Override public BytesRef term() { return br; } @@ -129,5 +130,18 @@ public class InstantiatedTermsEnum exten public Comparator getComparator() { return BytesRef.getUTF8SortedAsUnicodeComparator(); } + + @Override + public TermState termState() throws IOException { + final OrdTermState state = new OrdTermState(); + state.ord = upto - start; + return state; + } + + @Override + public SeekStatus seek(BytesRef term, TermState state) throws IOException { + assert state != null && state instanceof OrdTermState; + return seek(((OrdTermState)state).ord); // just use the ord for simplicity + } } Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestEmptyIndex.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestEmptyIndex.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestEmptyIndex.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestEmptyIndex.java Thu Jan 13 19:53:21 2011 @@ -22,6 +22,7 @@ import org.apache.lucene.analysis.MockAn import org.apache.lucene.index.IndexReader; import org.apache.lucene.index.IndexWriter; import org.apache.lucene.index.MultiFields; +import org.apache.lucene.index.MultiNorms; import org.apache.lucene.index.Term; import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.TermQuery; @@ -67,16 +68,9 @@ public class TestEmptyIndex extends Luce } private void testNorms(IndexReader r) throws IOException { - byte[] norms; - norms = r.norms("foo"); + byte[] norms = MultiNorms.norms(r, "foo"); if (norms != null) { assertEquals(0, norms.length); - norms = new byte[10]; - Arrays.fill(norms, (byte)10); - r.norms("foo", norms, 10); - for (byte b : norms) { - assertEquals((byte)10, b); - } } } Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestIndicesEquals.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestIndicesEquals.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestIndicesEquals.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestIndicesEquals.java Thu Jan 13 19:53:21 2011 @@ -30,6 +30,7 @@ import org.apache.lucene.document.Docume import org.apache.lucene.document.Field; import org.apache.lucene.index.IndexReader; import org.apache.lucene.index.IndexWriter; +import org.apache.lucene.index.MultiNorms; import org.apache.lucene.index.Payload; import org.apache.lucene.index.Term; import org.apache.lucene.index.DocsEnum; @@ -90,6 +91,10 @@ public class TestIndicesEquals extends L // create dir data IndexWriter indexWriter = new IndexWriter(dir, newIndexWriterConfig( TEST_VERSION_CURRENT, new MockAnalyzer())); + indexWriter.setInfoStream(VERBOSE ? System.out : null); + if (VERBOSE) { + System.out.println("TEST: make test index"); + } for (int i = 0; i < 500; i++) { Document document = new Document(); assembleDocument(document, i); @@ -319,6 +324,9 @@ public class TestIndicesEquals extends L protected void testEquals(Directory aprioriIndex, InstantiatedIndex testIndex) throws Exception { + if (VERBOSE) { + System.out.println("TEST: testEquals"); + } testTermDocsSomeMore(aprioriIndex, testIndex); IndexReader aprioriReader = IndexReader.open(aprioriIndex, false); @@ -341,8 +349,8 @@ public class TestIndicesEquals extends L // test norms as used by normal use - byte[] aprioriNorms = aprioriReader.norms((String) field); - byte[] testNorms = testReader.norms((String) field); + byte[] aprioriNorms = MultiNorms.norms(aprioriReader, (String) field); + byte[] testNorms = MultiNorms.norms(testReader, (String) field); if (aprioriNorms != null) { assertEquals(aprioriNorms.length, testNorms.length); @@ -350,35 +358,6 @@ public class TestIndicesEquals extends L for (int i = 0; i < aprioriNorms.length; i++) { assertEquals("norms does not equals for field " + field + " in document " + i, aprioriNorms[i], testNorms[i]); } - - // test norms as used by multireader - - aprioriNorms = new byte[aprioriReader.maxDoc()]; - aprioriReader.norms((String) field, aprioriNorms, 0); - - testNorms = new byte[testReader.maxDoc()]; - testReader.norms((String) field, testNorms, 0); - - assertEquals(aprioriNorms.length, testNorms.length); - - for (int i = 0; i < aprioriNorms.length; i++) { - assertEquals("norms does not equals for field " + field + " in document " + i, aprioriNorms[i], testNorms[i]); - } - - - // test norms as used by multireader - - aprioriNorms = new byte[aprioriReader.maxDoc() + 10]; - aprioriReader.norms((String) field, aprioriNorms, 10); - - testNorms = new byte[testReader.maxDoc() + 10]; - testReader.norms((String) field, testNorms, 10); - - assertEquals(aprioriNorms.length, testNorms.length); - - for (int i = 0; i < aprioriNorms.length; i++) { - assertEquals("norms does not equals for field " + field + " in document " + i, aprioriNorms[i], testNorms[i]); - } } } @@ -400,6 +379,9 @@ public class TestIndicesEquals extends L String aprioriField; while((aprioriField = aprioriFieldsEnum.next()) != null) { String testField = testFieldsEnum.next(); + if (VERBOSE) { + System.out.println("TEST: verify field=" + testField); + } assertEquals(aprioriField, testField); TermsEnum aprioriTermEnum = aprioriFieldsEnum.terms(); @@ -408,6 +390,9 @@ public class TestIndicesEquals extends L BytesRef aprioriText; while((aprioriText = aprioriTermEnum.next()) != null) { assertEquals(aprioriText, testTermEnum.next()); + if (VERBOSE) { + System.out.println("TEST: verify term=" + aprioriText.utf8ToString()); + } assertTrue(aprioriTermEnum.docFreq() == testTermEnum.docFreq()); @@ -433,6 +418,10 @@ public class TestIndicesEquals extends L assertEquals(DocsEnum.NO_MORE_DOCS, testTermDocs.nextDoc()); break; } + if (VERBOSE) { + System.out.println("TEST: verify doc=" + aprioriTermDocs.docID()); + } + assertTrue(testTermDocs.nextDoc() != DocsEnum.NO_MORE_DOCS); assertEquals(aprioriTermDocs.docID(), testTermDocs.docID()); @@ -444,12 +433,19 @@ public class TestIndicesEquals extends L DocsAndPositionsEnum aprioriTermPositions = aprioriTermEnum.docsAndPositions(MultiFields.getDeletedDocs(aprioriReader), null); DocsAndPositionsEnum testTermPositions = testTermEnum.docsAndPositions(MultiFields.getDeletedDocs(testReader), null); + if (VERBOSE) { + System.out.println("TEST: enum1=" + aprioriTermPositions + " enum2=" + testTermPositions); + } if (aprioriTermPositions != null) { for (int docIndex = 0; docIndex < aprioriReader.maxDoc(); docIndex++) { boolean hasNext = aprioriTermPositions.nextDoc() != DocsEnum.NO_MORE_DOCS; if (hasNext) { assertTrue(testTermPositions.nextDoc() != DocsEnum.NO_MORE_DOCS); + + if (VERBOSE) { + System.out.println("TEST: verify doc=" + aprioriTermPositions.docID()); + } assertEquals(aprioriTermPositions.freq(), testTermPositions.freq()); @@ -457,6 +453,10 @@ public class TestIndicesEquals extends L int aprioriPos = aprioriTermPositions.nextPosition(); int testPos = testTermPositions.nextPosition(); + if (VERBOSE) { + System.out.println("TEST: verify pos=" + aprioriPos); + } + assertEquals(aprioriPos, testPos); assertEquals(aprioriTermPositions.hasPayload(), testTermPositions.hasPayload()); Modified: lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestRealTime.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestRealTime.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestRealTime.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/instantiated/src/test/org/apache/lucene/store/instantiated/TestRealTime.java Thu Jan 13 19:53:21 2011 @@ -20,8 +20,8 @@ import org.apache.lucene.search.TermQuer import org.apache.lucene.search.Scorer; import org.apache.lucene.document.Document; import org.apache.lucene.document.Field; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.index.Term; -import org.apache.lucene.index.IndexReader; import org.apache.lucene.util.LuceneTestCase; /** @@ -67,7 +67,7 @@ public class TestRealTime extends Lucene @Override public void setScorer(Scorer scorer) {} @Override - public void setNextReader(IndexReader reader, int docBase) {} + public void setNextReader(AtomicReaderContext context) {} @Override public boolean acceptsDocsOutOfOrder() { return true; } @Override Modified: lucene/dev/branches/realtime_search/lucene/contrib/lucli/src/java/lucli/LuceneMethods.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/lucli/src/java/lucli/LuceneMethods.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/lucli/src/java/lucli/LuceneMethods.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/lucli/src/java/lucli/LuceneMethods.java Thu Jan 13 19:53:21 2011 @@ -41,6 +41,7 @@ import org.apache.lucene.analysis.tokena import org.apache.lucene.document.Document; import org.apache.lucene.document.Fieldable; import org.apache.lucene.index.IndexReader; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.index.IndexWriter; import org.apache.lucene.index.IndexWriterConfig; import org.apache.lucene.index.Fields; @@ -53,11 +54,10 @@ import org.apache.lucene.queryParser.Mul import org.apache.lucene.queryParser.ParseException; import org.apache.lucene.search.Collector; import org.apache.lucene.search.Explanation; -import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.Query; import org.apache.lucene.search.ScoreDoc; import org.apache.lucene.search.Scorer; -import org.apache.lucene.search.Searcher; +import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.store.FSDirectory; import org.apache.lucene.util.Version; import org.apache.lucene.util.BytesRef; @@ -73,7 +73,7 @@ class LuceneMethods { private List fields; //Fields as a vector private List indexedFields; //Fields as a vector private String fieldsArray[]; //Fields as an array - private Searcher searcher; + private IndexSearcher searcher; private Query query; //current query string private String analyzerClassFQN = null; // Analyzer class, if NULL, use default Analyzer @@ -233,7 +233,7 @@ class LuceneMethods { } @Override - public void setNextReader(IndexReader reader, int docBase) {} + public void setNextReader(AtomicReaderContext context) {} @Override public boolean acceptsDocsOutOfOrder() { return true; Modified: lucene/dev/branches/realtime_search/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/memory/src/java/org/apache/lucene/index/memory/MemoryIndex.java Thu Jan 13 19:53:21 2011 @@ -38,6 +38,9 @@ import org.apache.lucene.document.Docume import org.apache.lucene.document.FieldSelector; import org.apache.lucene.index.IndexReader; import org.apache.lucene.index.Fields; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; +import org.apache.lucene.index.OrdTermState; +import org.apache.lucene.index.TermState; import org.apache.lucene.index.Terms; import org.apache.lucene.index.TermsEnum; import org.apache.lucene.index.FieldsEnum; @@ -48,10 +51,10 @@ import org.apache.lucene.index.TermFreqV import org.apache.lucene.index.TermPositionVector; import org.apache.lucene.index.TermVectorMapper; import org.apache.lucene.index.FieldInvertState; +import org.apache.lucene.index.IndexReader.ReaderContext; import org.apache.lucene.search.Collector; import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.Query; -import org.apache.lucene.search.Searcher; import org.apache.lucene.search.Scorer; import org.apache.lucene.search.Similarity; import org.apache.lucene.store.RAMDirectory; // for javadocs @@ -421,7 +424,7 @@ public class MemoryIndex implements Seri if (query == null) throw new IllegalArgumentException("query must not be null"); - Searcher searcher = createSearcher(); + IndexSearcher searcher = createSearcher(); try { final float[] scores = new float[1]; // inits to 0.0f (no match) searcher.search(query, new Collector() { @@ -443,7 +446,7 @@ public class MemoryIndex implements Seri } @Override - public void setNextReader(IndexReader reader, int docBase) { } + public void setNextReader(AtomicReaderContext context) { } }); float score = scores[0]; return score; @@ -738,7 +741,8 @@ public class MemoryIndex implements Seri */ private final class MemoryIndexReader extends IndexReader { - private Searcher searcher; // needed to find searcher.getSimilarity() + private IndexSearcher searcher; // needed to find searcher.getSimilarity() + private final ReaderContext readerInfos = new AtomicReaderContext(this); private MemoryIndexReader() { super(); // avoid as much superclass baggage as possible @@ -765,6 +769,11 @@ public class MemoryIndex implements Seri if (DEBUG) System.err.println("MemoryIndexReader.docFreq: " + term + ", freq:" + freq); return freq; } + + @Override + public ReaderContext getTopReaderContext() { + return readerInfos; + } @Override public Fields fields() { @@ -877,10 +886,6 @@ public class MemoryIndex implements Seri } @Override - public void cacheCurrentTerm() { - } - - @Override public long ord() { return termUpto; } @@ -910,8 +915,21 @@ public class MemoryIndex implements Seri public Comparator getComparator() { return BytesRef.getUTF8SortedAsUnicodeComparator(); } - } + @Override + public SeekStatus seek(BytesRef term, TermState state) throws IOException { + assert state != null; + return this.seek(((OrdTermState)state).ord); + } + + @Override + public TermState termState() throws IOException { + OrdTermState ts = new OrdTermState(); + ts.ord = termUpto; + return ts; + } + } + private class MemoryDocsEnum extends DocsEnum { private ArrayIntList positions; private boolean hasNext; @@ -1135,7 +1153,7 @@ public class MemoryIndex implements Seri return Similarity.getDefault(); } - private void setSearcher(Searcher searcher) { + private void setSearcher(IndexSearcher searcher) { this.searcher = searcher; } @@ -1168,13 +1186,6 @@ public class MemoryIndex implements Seri } @Override - public void norms(String fieldName, byte[] bytes, int offset) { - if (DEBUG) System.err.println("MemoryIndexReader.norms*: " + fieldName); - byte[] norms = norms(fieldName); - System.arraycopy(norms, 0, bytes, offset, norms.length); - } - - @Override protected void doSetNorm(int doc, String fieldName, byte value) { throw new UnsupportedOperationException(); } Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/index/FieldNormModifier.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/index/FieldNormModifier.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/index/FieldNormModifier.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/index/FieldNormModifier.java Thu Jan 13 19:53:21 2011 @@ -33,7 +33,7 @@ import org.apache.lucene.util.ReaderUtil /** * Given a directory and a list of fields, updates the fieldNorms in place for every document. * - * If Similarity class is specified, uses its lengthNorm method to set norms. + * If Similarity class is specified, uses its computeNorm method to set norms. * If -n command line argument is used, removed field norms, as if * {@link org.apache.lucene.document.Field.Index}.NO_NORMS was used. * @@ -119,6 +119,7 @@ public class FieldNormModifier { final List subReaders = new ArrayList(); ReaderUtil.gatherSubReaders(subReaders, reader); + final FieldInvertState invertState = new FieldInvertState(); for(IndexReader subReader : subReaders) { final Bits delDocs = subReader.getDeletedDocs(); @@ -143,9 +144,11 @@ public class FieldNormModifier { } } + invertState.setBoost(1.0f); for (int d = 0; d < termCounts.length; d++) { if (delDocs == null || !delDocs.get(d)) { - subReader.setNorm(d, fieldName, sim.encodeNormValue(sim.lengthNorm(fieldName, termCounts[d]))); + invertState.setLength(termCounts[d]); + subReader.setNorm(d, fieldName, sim.encodeNormValue(sim.computeNorm(fieldName, invertState))); } } } Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/java/org/apache/lucene/misc/SweetSpotSimilarity.java Thu Jan 13 19:53:21 2011 @@ -146,7 +146,7 @@ public class SweetSpotSimilarity extends else numTokens = state.getLength(); - return state.getBoost() * lengthNorm(fieldName, numTokens); + return state.getBoost() * computeLengthNorm(fieldName, numTokens); } /** @@ -167,8 +167,7 @@ public class SweetSpotSimilarity extends * * @see #setLengthNormFactors */ - @Override - public float lengthNorm(String fieldName, int numTerms) { + public float computeLengthNorm(String fieldName, int numTerms) { int l = ln_min; int h = ln_max; float s = ln_steep; Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/index/TestFieldNormModifier.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/index/TestFieldNormModifier.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/index/TestFieldNormModifier.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/index/TestFieldNormModifier.java Thu Jan 13 19:53:21 2011 @@ -23,6 +23,7 @@ import java.util.Arrays; import org.apache.lucene.analysis.MockAnalyzer; import org.apache.lucene.document.Document; import org.apache.lucene.document.Field; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.search.Collector; import org.apache.lucene.search.DefaultSimilarity; import org.apache.lucene.search.IndexSearcher; @@ -43,8 +44,8 @@ public class TestFieldNormModifier exten /** inverts the normal notion of lengthNorm */ public static Similarity s = new DefaultSimilarity() { @Override - public float lengthNorm(String fieldName, int numTokens) { - return numTokens; + public float computeNorm(String fieldName, FieldInvertState state) { + return state.getBoost() * (discountOverlaps ? state.getLength() - state.getNumOverlap() : state.getLength()); } }; @@ -84,7 +85,7 @@ public class TestFieldNormModifier exten public void testFieldWithNoNorm() throws Exception { IndexReader r = IndexReader.open(store, false); - byte[] norms = r.norms("nonorm"); + byte[] norms = MultiNorms.norms(r, "nonorm"); // sanity check, norms should all be 1 assertTrue("Whoops we have norms?", !r.hasNorms("nonorm")); @@ -98,7 +99,7 @@ public class TestFieldNormModifier exten // nothing should have changed r = IndexReader.open(store, false); - norms = r.norms("nonorm"); + norms = MultiNorms.norms(r, "nonorm"); assertTrue("Whoops we have norms?", !r.hasNorms("nonorm")); assertNull(norms); @@ -122,8 +123,8 @@ public class TestFieldNormModifier exten scores[doc + docBase] = scorer.score(); } @Override - public void setNextReader(IndexReader reader, int docBase) { - this.docBase = docBase; + public void setNextReader(AtomicReaderContext context) { + docBase = context.docBase; } @Override public void setScorer(Scorer scorer) throws IOException { @@ -157,8 +158,8 @@ public class TestFieldNormModifier exten scores[doc + docBase] = scorer.score(); } @Override - public void setNextReader(IndexReader reader, int docBase) { - this.docBase = docBase; + public void setNextReader(AtomicReaderContext context) { + docBase = context.docBase; } @Override public void setScorer(Scorer scorer) throws IOException { @@ -183,14 +184,14 @@ public class TestFieldNormModifier exten public void testNormKiller() throws IOException { IndexReader r = IndexReader.open(store, false); - byte[] oldNorms = r.norms("untokfield"); + byte[] oldNorms = MultiNorms.norms(r, "untokfield"); r.close(); FieldNormModifier fnm = new FieldNormModifier(store, s); fnm.reSetNorms("untokfield"); r = IndexReader.open(store, false); - byte[] newNorms = r.norms("untokfield"); + byte[] newNorms = MultiNorms.norms(r, "untokfield"); r.close(); assertFalse(Arrays.equals(oldNorms, newNorms)); @@ -209,8 +210,8 @@ public class TestFieldNormModifier exten scores[doc + docBase] = scorer.score(); } @Override - public void setNextReader(IndexReader reader, int docBase) { - this.docBase = docBase; + public void setNextReader(AtomicReaderContext context) { + docBase = context.docBase; } @Override public void setScorer(Scorer scorer) throws IOException { Modified: lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/misc/src/test/org/apache/lucene/misc/SweetSpotSimilarityTest.java Thu Jan 13 19:53:21 2011 @@ -21,13 +21,14 @@ package org.apache.lucene.misc; import org.apache.lucene.search.DefaultSimilarity; import org.apache.lucene.search.Similarity; import org.apache.lucene.util.LuceneTestCase; +import org.apache.lucene.index.FieldInvertState; /** * Test of the SweetSpotSimilarity */ public class SweetSpotSimilarityTest extends LuceneTestCase { - public void testSweetSpotLengthNorm() { + public void testSweetSpotComputeNorm() { SweetSpotSimilarity ss = new SweetSpotSimilarity(); ss.setLengthNormFactors(1,1,0.5f); @@ -37,10 +38,13 @@ public class SweetSpotSimilarityTest ext // base case, should degrade - + final FieldInvertState invertState = new FieldInvertState(); + invertState.setBoost(1.0f); for (int i = 1; i < 1000; i++) { + invertState.setLength(i); assertEquals("base case: i="+i, - d.lengthNorm("foo",i), s.lengthNorm("foo",i), + d.computeNorm("foo", invertState), + s.computeNorm("foo", invertState), 0.0f); } @@ -49,14 +53,21 @@ public class SweetSpotSimilarityTest ext ss.setLengthNormFactors(3,10,0.5f); for (int i = 3; i <=10; i++) { + invertState.setLength(i); assertEquals("3,10: spot i="+i, - 1.0f, s.lengthNorm("foo",i), + 1.0f, + s.computeNorm("foo", invertState), 0.0f); } for (int i = 10; i < 1000; i++) { + invertState.setLength(i-9); + final float normD = d.computeNorm("foo", invertState); + invertState.setLength(i); + final float normS = s.computeNorm("foo", invertState); assertEquals("3,10: 10 notFilters = null; ArrayList mustFilters = null; - private DocIdSetIterator getDISI(ArrayList filters, int index, IndexReader reader) + private DocIdSetIterator getDISI(ArrayList filters, int index, AtomicReaderContext context) throws IOException { - return filters.get(index).getDocIdSet(reader).iterator(); + return filters.get(index).getDocIdSet(context).iterator(); } /** @@ -52,21 +53,21 @@ public class BooleanFilter extends Filte * of the filters that have been added. */ @Override - public DocIdSet getDocIdSet(IndexReader reader) throws IOException + public DocIdSet getDocIdSet(AtomicReaderContext context) throws IOException { OpenBitSetDISI res = null; - + final IndexReader reader = context.reader; if (shouldFilters != null) { for (int i = 0; i < shouldFilters.size(); i++) { if (res == null) { - res = new OpenBitSetDISI(getDISI(shouldFilters, i, reader), reader.maxDoc()); + res = new OpenBitSetDISI(getDISI(shouldFilters, i, context), reader.maxDoc()); } else { - DocIdSet dis = shouldFilters.get(i).getDocIdSet(reader); + DocIdSet dis = shouldFilters.get(i).getDocIdSet(context); if(dis instanceof OpenBitSet) { // optimized case for OpenBitSets res.or((OpenBitSet) dis); } else { - res.inPlaceOr(getDISI(shouldFilters, i, reader)); + res.inPlaceOr(getDISI(shouldFilters, i, context)); } } } @@ -75,15 +76,15 @@ public class BooleanFilter extends Filte if (notFilters!=null) { for (int i = 0; i < notFilters.size(); i++) { if (res == null) { - res = new OpenBitSetDISI(getDISI(notFilters, i, reader), reader.maxDoc()); + res = new OpenBitSetDISI(getDISI(notFilters, i, context), reader.maxDoc()); res.flip(0, reader.maxDoc()); // NOTE: may set bits on deleted docs } else { - DocIdSet dis = notFilters.get(i).getDocIdSet(reader); + DocIdSet dis = notFilters.get(i).getDocIdSet(context); if(dis instanceof OpenBitSet) { // optimized case for OpenBitSets res.andNot((OpenBitSet) dis); } else { - res.inPlaceNot(getDISI(notFilters, i, reader)); + res.inPlaceNot(getDISI(notFilters, i, context)); } } } @@ -92,14 +93,14 @@ public class BooleanFilter extends Filte if (mustFilters!=null) { for (int i = 0; i < mustFilters.size(); i++) { if (res == null) { - res = new OpenBitSetDISI(getDISI(mustFilters, i, reader), reader.maxDoc()); + res = new OpenBitSetDISI(getDISI(mustFilters, i, context), reader.maxDoc()); } else { - DocIdSet dis = mustFilters.get(i).getDocIdSet(reader); + DocIdSet dis = mustFilters.get(i).getDocIdSet(context); if(dis instanceof OpenBitSet) { // optimized case for OpenBitSets res.and((OpenBitSet) dis); } else { - res.inPlaceAnd(getDISI(mustFilters, i, reader)); + res.inPlaceAnd(getDISI(mustFilters, i, context)); } } } Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BoostingQuery.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BoostingQuery.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BoostingQuery.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/BoostingQuery.java Thu Jan 13 19:53:21 2011 @@ -23,7 +23,7 @@ import org.apache.lucene.index.IndexRead import org.apache.lucene.search.BooleanQuery; import org.apache.lucene.search.DefaultSimilarity; import org.apache.lucene.search.Query; -import org.apache.lucene.search.Searcher; +import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.Similarity; /** * The BoostingQuery class can be used to effectively demote results that match a given query. @@ -58,7 +58,7 @@ public class BoostingQuery extends Query BooleanQuery result = new BooleanQuery() { @Override - public Similarity getSimilarity(Searcher searcher) { + public Similarity getSimilarity(IndexSearcher searcher) { return new DefaultSimilarity() { @Override Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/ChainedFilter.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/ChainedFilter.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/ChainedFilter.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/ChainedFilter.java Thu Jan 13 19:53:21 2011 @@ -20,6 +20,7 @@ package org.apache.lucene.search; import java.io.IOException; import org.apache.lucene.index.IndexReader; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.search.DocIdSet; import org.apache.lucene.search.DocIdSetIterator; import org.apache.lucene.search.Filter; @@ -96,21 +97,21 @@ public class ChainedFilter extends Filte * {@link Filter#getDocIdSet}. */ @Override - public DocIdSet getDocIdSet(IndexReader reader) throws IOException + public DocIdSet getDocIdSet(AtomicReaderContext context) throws IOException { int[] index = new int[1]; // use array as reference to modifiable int; index[0] = 0; // an object attribute would not be thread safe. if (logic != -1) - return getDocIdSet(reader, logic, index); + return getDocIdSet(context, logic, index); else if (logicArray != null) - return getDocIdSet(reader, logicArray, index); + return getDocIdSet(context, logicArray, index); else - return getDocIdSet(reader, DEFAULT, index); + return getDocIdSet(context, DEFAULT, index); } - private DocIdSetIterator getDISI(Filter filter, IndexReader reader) + private DocIdSetIterator getDISI(Filter filter, AtomicReaderContext context) throws IOException { - DocIdSet docIdSet = filter.getDocIdSet(reader); + DocIdSet docIdSet = filter.getDocIdSet(context); if (docIdSet == null) { return DocIdSet.EMPTY_DOCIDSET.iterator(); } else { @@ -123,9 +124,10 @@ public class ChainedFilter extends Filte } } - private OpenBitSetDISI initialResult(IndexReader reader, int logic, int[] index) + private OpenBitSetDISI initialResult(AtomicReaderContext context, int logic, int[] index) throws IOException { + IndexReader reader = context.reader; OpenBitSetDISI result; /** * First AND operation takes place against a completely false @@ -133,12 +135,12 @@ public class ChainedFilter extends Filte */ if (logic == AND) { - result = new OpenBitSetDISI(getDISI(chain[index[0]], reader), reader.maxDoc()); + result = new OpenBitSetDISI(getDISI(chain[index[0]], context), reader.maxDoc()); ++index[0]; } else if (logic == ANDNOT) { - result = new OpenBitSetDISI(getDISI(chain[index[0]], reader), reader.maxDoc()); + result = new OpenBitSetDISI(getDISI(chain[index[0]], context), reader.maxDoc()); result.flip(0,reader.maxDoc()); // NOTE: may set bits for deleted docs. ++index[0]; } @@ -155,13 +157,13 @@ public class ChainedFilter extends Filte * @param logic Logical operation * @return DocIdSet */ - private DocIdSet getDocIdSet(IndexReader reader, int logic, int[] index) + private DocIdSet getDocIdSet(AtomicReaderContext context, int logic, int[] index) throws IOException { - OpenBitSetDISI result = initialResult(reader, logic, index); + OpenBitSetDISI result = initialResult(context, logic, index); for (; index[0] < chain.length; index[0]++) { - doChain(result, logic, chain[index[0]].getDocIdSet(reader)); + doChain(result, logic, chain[index[0]].getDocIdSet(context)); } return result; } @@ -172,16 +174,16 @@ public class ChainedFilter extends Filte * @param logic Logical operation * @return DocIdSet */ - private DocIdSet getDocIdSet(IndexReader reader, int[] logic, int[] index) + private DocIdSet getDocIdSet(AtomicReaderContext info, int[] logic, int[] index) throws IOException { if (logic.length != chain.length) throw new IllegalArgumentException("Invalid number of elements in logic array"); - OpenBitSetDISI result = initialResult(reader, logic[0], index); + OpenBitSetDISI result = initialResult(info, logic[0], index); for (; index[0] < chain.length; index[0]++) { - doChain(result, logic[index[0]], chain[index[0]].getDocIdSet(reader)); + doChain(result, logic[index[0]], chain[index[0]].getDocIdSet(info)); } return result; } Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/DuplicateFilter.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/DuplicateFilter.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/DuplicateFilter.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/DuplicateFilter.java Thu Jan 13 19:53:21 2011 @@ -19,6 +19,7 @@ import java.io.IOException; import org.apache.lucene.index.IndexReader; import org.apache.lucene.util.BytesRef; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.index.Terms; import org.apache.lucene.index.DocsEnum; import org.apache.lucene.index.TermsEnum; @@ -27,7 +28,8 @@ import org.apache.lucene.util.OpenBitSet import org.apache.lucene.util.Bits; public class DuplicateFilter extends Filter -{ +{ // TODO: make duplicate filter aware of ReaderContext such that we can + // filter duplicates across segments String fieldName; @@ -70,15 +72,15 @@ public class DuplicateFilter extends Fil } @Override - public DocIdSet getDocIdSet(IndexReader reader) throws IOException + public DocIdSet getDocIdSet(AtomicReaderContext context) throws IOException { if(processingMode==PM_FAST_INVALIDATION) { - return fastBits(reader); + return fastBits(context.reader); } else { - return correctBits(reader); + return correctBits(context.reader); } } @@ -96,7 +98,7 @@ public class DuplicateFilter extends Fil } else { docs = termsEnum.docs(delDocs, docs); int doc = docs.nextDoc(); - if (doc != docs.NO_MORE_DOCS) { + if (doc != DocsEnum.NO_MORE_DOCS) { if (keepMode == KM_USE_FIRST_OCCURRENCE) { bits.set(doc); } else { @@ -104,7 +106,7 @@ public class DuplicateFilter extends Fil while (true) { lastDoc = doc; doc = docs.nextDoc(); - if (doc == docs.NO_MORE_DOCS) { + if (doc == DocsEnum.NO_MORE_DOCS) { break; } } @@ -136,7 +138,7 @@ public class DuplicateFilter extends Fil // unset potential duplicates docs = termsEnum.docs(delDocs, docs); int doc = docs.nextDoc(); - if (doc != docs.NO_MORE_DOCS) { + if (doc != DocsEnum.NO_MORE_DOCS) { if (keepMode == KM_USE_FIRST_OCCURRENCE) { doc = docs.nextDoc(); } @@ -147,7 +149,7 @@ public class DuplicateFilter extends Fil lastDoc = doc; bits.clear(lastDoc); doc = docs.nextDoc(); - if (doc == docs.NO_MORE_DOCS) { + if (doc == DocsEnum.NO_MORE_DOCS) { break; } } Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/FuzzyLikeThisQuery.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/FuzzyLikeThisQuery.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/FuzzyLikeThisQuery.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/FuzzyLikeThisQuery.java Thu Jan 13 19:53:21 2011 @@ -292,7 +292,7 @@ public class FuzzyLikeThisQuery extends { //optimize where only one selected variant ScoreTerm st= variants.get(0); - TermQuery tq = new FuzzyTermQuery(st.term,ignoreTF); + Query tq = ignoreTF ? new ConstantScoreQuery(new TermQuery(st.term)) : new TermQuery(st.term, 1); tq.setBoost(st.score); // set the boost to a mix of IDF and score bq.add(tq, BooleanClause.Occur.SHOULD); } @@ -303,7 +303,8 @@ public class FuzzyLikeThisQuery extends .hasNext();) { ScoreTerm st = iterator2.next(); - TermQuery tq = new FuzzyTermQuery(st.term,ignoreTF); // found a match + // found a match + Query tq = ignoreTF ? new ConstantScoreQuery(new TermQuery(st.term)) : new TermQuery(st.term, 1); tq.setBoost(st.score); // set the boost using the ScoreTerm's score termVariants.add(tq, BooleanClause.Occur.SHOULD); // add to query } @@ -348,45 +349,8 @@ public class FuzzyLikeThisQuery extends return termA.score < termB.score; } - } - - //overrides basic TermQuery to negate effects of IDF (idf is factored into boost of containing BooleanQuery) - private static class FuzzyTermQuery extends TermQuery - { - boolean ignoreTF; - public FuzzyTermQuery(Term t, boolean ignoreTF) - { - super(t); - this.ignoreTF=ignoreTF; - } - @Override - public Similarity getSimilarity(Searcher searcher) - { - Similarity result = super.getSimilarity(searcher); - result = new SimilarityDelegator(result) { - - @Override - public float tf(float freq) - { - if(ignoreTF) - { - return 1; //ignore tf - } - return super.tf(freq); - } - @Override - public float idf(int docFreq, int numDocs) - { - //IDF is already factored into individual term boosts - return 1; - } - }; - return result; - } - } + } - - /* (non-Javadoc) * @see org.apache.lucene.search.Query#toString(java.lang.String) */ Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/TermsFilter.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/TermsFilter.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/TermsFilter.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/java/org/apache/lucene/search/TermsFilter.java Thu Jan 13 19:53:21 2011 @@ -23,6 +23,7 @@ import java.util.Set; import java.util.TreeSet; import org.apache.lucene.index.IndexReader; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.index.Term; import org.apache.lucene.index.DocsEnum; import org.apache.lucene.index.Terms; @@ -57,7 +58,8 @@ public class TermsFilter extends Filter * @see org.apache.lucene.search.Filter#getDocIdSet(org.apache.lucene.index.IndexReader) */ @Override - public DocIdSet getDocIdSet(IndexReader reader) throws IOException { + public DocIdSet getDocIdSet(AtomicReaderContext context) throws IOException { + IndexReader reader = context.reader; OpenBitSet result=new OpenBitSet(reader.maxDoc()); Fields fields = reader.fields(); BytesRef br = new BytesRef(); Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/BooleanFilterTest.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/BooleanFilterTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/BooleanFilterTest.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/BooleanFilterTest.java Thu Jan 13 19:53:21 2011 @@ -24,6 +24,7 @@ import org.apache.lucene.analysis.MockTo import org.apache.lucene.document.Document; import org.apache.lucene.document.Field; import org.apache.lucene.index.IndexReader; +import org.apache.lucene.index.IndexReader.AtomicReaderContext; import org.apache.lucene.index.RandomIndexWriter; import org.apache.lucene.index.SlowMultiReaderWrapper; import org.apache.lucene.index.Term; @@ -83,7 +84,7 @@ public class BooleanFilterTest extends L private void tstFilterCard(String mes, int expected, Filter filt) throws Throwable { - DocIdSetIterator disi = filt.getDocIdSet(reader).iterator(); + DocIdSetIterator disi = filt.getDocIdSet(new AtomicReaderContext(reader)).iterator(); int actual = 0; while (disi.nextDoc() != DocIdSetIterator.NO_MORE_DOCS) { actual++; Modified: lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/ChainedFilterTest.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/ChainedFilterTest.java?rev=1058718&r1=1058717&r2=1058718&view=diff ============================================================================== --- lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/ChainedFilterTest.java (original) +++ lucene/dev/branches/realtime_search/lucene/contrib/queries/src/test/org/apache/lucene/search/ChainedFilterTest.java Thu Jan 13 19:53:21 2011 @@ -29,11 +29,10 @@ import org.apache.lucene.search.BooleanC import org.apache.lucene.search.BooleanQuery; import org.apache.lucene.search.CachingWrapperFilter; import org.apache.lucene.search.Filter; -import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.MatchAllDocsQuery; import org.apache.lucene.search.Query; import org.apache.lucene.search.QueryWrapperFilter; -import org.apache.lucene.search.Searcher; +import org.apache.lucene.search.IndexSearcher; import org.apache.lucene.search.TermQuery; import org.apache.lucene.search.TermRangeFilter; import org.apache.lucene.search.TopDocs; @@ -195,7 +194,7 @@ public class ChainedFilterTest extends L IndexReader reader = writer.getReader(); writer.close(); - Searcher searcher = new IndexSearcher(reader); + IndexSearcher searcher = new IndexSearcher(reader); Query query = new TermQuery(new Term("none", "none"));