lucene-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rm...@apache.org
Subject svn commit: r1169470 [3/4] - in /lucene/dev/trunk: lucene/ lucene/contrib/instantiated/src/java/org/apache/lucene/store/instantiated/ lucene/contrib/memory/src/java/org/apache/lucene/index/memory/ lucene/contrib/misc/src/java/org/apache/lucene/index/ l...
Date Sun, 11 Sep 2011 15:47:25 GMT
Added: lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/similarities/package.html
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/similarities/package.html?rev=1169470&view=auto
==============================================================================
--- lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/similarities/package.html (added)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/similarities/package.html Sun Sep 11 15:47:21 2011
@@ -0,0 +1,174 @@
+<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements.  See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License.  You may obtain a copy of the License at
+
+     http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<html>
+<head>
+   <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
+</head>
+<body>
+This package contains the various ranking models that can be used in Lucene. The
+abstract class {@link org.apache.lucene.search.similarities.Similarity} serves
+as the base for ranking functions. For searching, users can employ the models
+already implemented or create their own by extending one of the classes in this
+package.
+
+<h2>Table Of Contents</h2>
+<p>
+    <ol>
+        <li><a href="#sims">Summary of the Ranking Methods</a></li>
+        <li><a href="#providers">Similarity Providers<a/></li>
+        <li><a href="#changingSimilarity">Changing the Similarity</a></li>
+    </ol>
+</p>
+
+
+<a name="sims"></a>
+<h2>Summary of the Ranking Methods</h2>
+
+<p>{@link org.apache.lucene.search.similarities.DefaultSimilarity} is the original Lucene
+scoring function. It is based on a highly optimized Vector Space Model. For more
+information, see {@link org.apache.lucene.search.similarities.TFIDFSimilarity}.</p>
+
+<p>{@link org.apache.lucene.search.similarities.BM25Similarity} is an optimized
+implementation of the successful Okapi BM25 model.</p>
+
+<p>{@link org.apache.lucene.search.similarities.SimilarityBase} provides a basic
+implementation of the Similarity contract and exposes a highly simplified
+interface, which makes it an ideal starting point for new ranking functions.
+Lucene ships the following methods built on
+{@link org.apache.lucene.search.similarities.SimilarityBase}:
+
+<a name="framework"></a>
+<ul>
+  <li>Amati and Rijsbergen's {@linkplain org.apache.lucene.search.similarities.DFRSimilarity DFR} framework;</li>
+  <li>Clinchant and Gaussier's {@linkplain org.apache.lucene.search.similarities.IBSimilarity Information-based models}
+    for IR;</li>
+  <li>The implementation of two {@linkplain org.apache.lucene.search.similarities.LMSimilarity language models} from
+  Zhai and Lafferty's paper.</li>
+</ul>
+
+Since {@link org.apache.lucene.search.similarities.SimilarityBase} is not
+optimized to the same extent as
+{@link org.apache.lucene.search.similarities.DefaultSimilarity} and
+{@link org.apache.lucene.search.similarities.BM25Similarity}, a difference in
+performance is to be expected when using the methods listed above. However,
+optimizations can always be implemented in subclasses; see
+<a href="#changingSimilarity">below</a>.</p>
+
+
+<a name="providers"></a>
+<h2>Similarity Providers</h2>
+
+<p>{@link org.apache.lucene.search.similarities.SimilarityProvider}s are factories
+that return Similarities per-field and compute coordination factors and normalization 
+values for the query.
+{@link org.apache.lucene.search.similarities.DefaultSimilarityProvider} is the
+default implementation used by Lucene, geared towards vector-spaced search: it returns 
+{@link org.apache.lucene.search.similarities.DefaultSimilarity} for every field,
+and implements coordination-level matching and query normalization. 
+{@link org.apache.lucene.search.similarities.BasicSimilarityProvider} is geared towards
+non-vector-space models and does not implement coordination-level matching or query
+normalization. It is a convenience implementation that returns an arbitrary 
+{@link org.apache.lucene.search.similarities.Similarity} for every field.
+You can write your own SimilarityProvider to return different Similarities for different
+fields: for example you might want to use different parameter values for different fields,
+or maybe even entirely different ranking algorithms.
+</p>
+
+
+<a name="changingSimilarity"></a>
+<h2>Changing Similarity</h2>
+
+<p>Chances are the available Similarities are sufficient for all
+    your searching needs.
+    However, in some applications it may be necessary to customize your <a
+        href="Similarity.html">Similarity</a> implementation. For instance, some
+    applications do not need to
+    distinguish between shorter and longer documents (see <a
+        href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">a "fair" similarity</a>).</p>
+
+<p>To change {@link org.apache.lucene.search.similarities.Similarity}, one must do so for both indexing and
+    searching, and the changes must happen before
+    either of these actions take place. Although in theory there is nothing stopping you from changing mid-stream, it
+    just isn't well-defined what is going to happen.
+</p>
+
+<p>To make this change, implement your own {@link org.apache.lucene.search.similarities.Similarity} (likely
+    you'll want to simply subclass an existing method, be it
+    {@link org.apache.lucene.search.similarities.DefaultSimilarity} or a descendant of
+    {@link org.apache.lucene.search.similarities.SimilarityBase}) and
+    {@link org.apache.lucene.search.similarities.SimilarityProvider} (or use
+    {@link org.apache.lucene.search.similarities.BasicSimilarityProvider}), and
+    then register the new class by calling
+    {@link org.apache.lucene.index.IndexWriterConfig#setSimilarityProvider(SimilarityProvider)}
+    before indexing and
+    {@link org.apache.lucene.search.IndexSearcher#setSimilarityProvider(SimilarityProvider)}
+    before searching.
+</p>
+
+<h3>Extending {@linkplain org.apache.lucene.search.similarities.SimilarityBase}</h3>
+<p>
+The easiest way to quickly implement a new ranking method is to extend
+{@link org.apache.lucene.search.similarities.SimilarityBase}, which provides
+basic implementations for the low level . Subclasses are only required to
+implement the {@link org.apache.lucene.search.similarities.SimilarityBase#score(BasicStats, float, float)}
+and {@link org.apache.lucene.search.similarities.SimilarityBase#toString()}
+methods.</p>
+
+<p>Another options is to extend one of the <a href="#framework">frameworks</a>
+based on {@link org.apache.lucene.search.similarities.SimilarityBase}. These
+Similarities are implemented modularly, e.g.
+{@link org.apache.lucene.search.similarities.DFRSimilarity} delegates
+computation of the three parts of its formula to the classes
+{@link org.apache.lucene.search.similarities.BasicModel},
+{@link org.apache.lucene.search.similarities.AfterEffect} and
+{@link org.apache.lucene.search.similarities.Normalization}. Instead of
+subclassing the Similarity, one can simply introduce a new basic model and tell
+{@link org.apache.lucene.search.similarities.DFRSimilarity} to use it.</p>
+
+<h3>Changing {@linkplain org.apache.lucene.search.similarities.DefaultSimilarity}</h3>
+<p>
+    If you are interested in use cases for changing your similarity, see the Lucene users's mailing list at <a
+        href="http://www.nabble.com/Overriding-Similarity-tf2128934.html">Overriding Similarity</a>.
+    In summary, here are a few use cases:
+    <ol>
+        <li><p>The <code>SweetSpotSimilarity</code> in
+            <code>org.apache.lucene.misc</code> gives small
+            increases as the frequency increases a small amount
+            and then greater increases when you hit the "sweet spot", i.e. where
+            you think the frequency of terms is more significant.</p></li>
+        <li><p>Overriding tf &mdash; In some applications, it doesn't matter what the score of a document is as long as a
+            matching term occurs. In these
+            cases people have overridden Similarity to return 1 from the tf() method.</p></li>
+        <li><p>Changing Length Normalization &mdash; By overriding
+            {@link org.apache.lucene.search.similarities.Similarity#computeNorm(FieldInvertState state)},
+            it is possible to discount how the length of a field contributes
+            to a score. In {@link org.apache.lucene.search.similarities.DefaultSimilarity},
+            lengthNorm = 1 / (numTerms in field)^0.5, but if one changes this to be
+            1 / (numTerms in field), all fields will be treated
+            <a href="http://www.gossamer-threads.com/lists/lucene/java-user/38967#38967">"fairly"</a>.</p></li>
+    </ol>
+    In general, Chris Hostetter sums it up best in saying (from <a
+        href="http://www.gossamer-threads.com/lists/lucene/java-user/39125#39125">the Lucene users's mailing list</a>):
+    <blockquote>[One would override the Similarity in] ... any situation where you know more about your data then just
+        that
+        it's "text" is a situation where it *might* make sense to to override your
+        Similarity method.</blockquote>
+</p>
+
+</body>
+</html>

Modified: lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/spans/SpanScorer.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/spans/SpanScorer.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/spans/SpanScorer.java (original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/spans/SpanScorer.java Sun Sep 11 15:47:21 2011
@@ -19,11 +19,9 @@ package org.apache.lucene.search.spans;
 
 import java.io.IOException;
 
-import org.apache.lucene.search.Explanation;
-import org.apache.lucene.search.TFIDFSimilarity;
 import org.apache.lucene.search.Weight;
 import org.apache.lucene.search.Scorer;
-import org.apache.lucene.search.Similarity;
+import org.apache.lucene.search.similarities.Similarity;
 
 /**
  * Public for extension only.

Modified: lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/spans/SpanWeight.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/spans/SpanWeight.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/spans/SpanWeight.java (original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/search/spans/SpanWeight.java Sun Sep 11 15:47:21 2011
@@ -21,7 +21,8 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.index.IndexReader.ReaderContext;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.search.*;
-import org.apache.lucene.search.Similarity.SloppyDocScorer;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.Similarity.SloppyDocScorer;
 import org.apache.lucene.util.TermContext;
 
 import java.io.IOException;

Modified: lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/index/DocHelper.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/index/DocHelper.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/index/DocHelper.java (original)
+++ lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/index/DocHelper.java Sun Sep 11 15:47:21 2011
@@ -33,7 +33,7 @@ import org.apache.lucene.document.String
 import org.apache.lucene.index.FieldInfo.IndexOptions;
 import org.apache.lucene.document.FieldType;
 import org.apache.lucene.document.TextField;
-import org.apache.lucene.search.SimilarityProvider;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 
 import static org.apache.lucene.util.LuceneTestCase.TEST_VERSION_CURRENT;

Modified: lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/search/CheckHits.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/search/CheckHits.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/search/CheckHits.java (original)
+++ lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/search/CheckHits.java Sun Sep 11 15:47:21 2011
@@ -18,6 +18,7 @@ package org.apache.lucene.search;
  */
 
 import java.io.IOException;
+import java.util.Locale;
 import java.util.Set;
 import java.util.TreeSet;
 import java.util.Random;
@@ -35,7 +36,7 @@ public class CheckHits {
    * different  order of operations from the actual scoring method ...
    * this allows for a small amount of variation
    */
-  public static float EXPLAIN_SCORE_TOLERANCE_DELTA = 0.0002f;
+  public static float EXPLAIN_SCORE_TOLERANCE_DELTA = 0.02f;
     
   /**
    * Tests that all documents up to maxDoc which are *not* in the
@@ -327,6 +328,10 @@ public class CheckHits {
     if (!deep) return;
 
     Explanation detail[] = expl.getDetails();
+    // TODO: can we improve this entire method? its really geared to work only with TF/IDF
+    if (expl.getDescription().endsWith("computed from:")) {
+      return; // something more complicated.
+    }
     if (detail!=null) {
       if (detail.length==1) {
         // simple containment, unless its a freq of: (which lets a query explain how the freq is calculated), 
@@ -338,7 +343,7 @@ public class CheckHits {
         // - end with one of: "product of:", "sum of:", "max of:", or
         // - have "max plus <x> times others" (where <x> is float).
         float x = 0;
-        String descr = expl.getDescription().toLowerCase();
+        String descr = expl.getDescription().toLowerCase(Locale.ENGLISH);
         boolean productOf = descr.endsWith("product of:");
         boolean sumOf = descr.endsWith("sum of:");
         boolean maxOf = descr.endsWith("max of:");

Added: lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/search/RandomSimilarityProvider.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/search/RandomSimilarityProvider.java?rev=1169470&view=auto
==============================================================================
--- lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/search/RandomSimilarityProvider.java (added)
+++ lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/search/RandomSimilarityProvider.java Sun Sep 11 15:47:21 2011
@@ -0,0 +1,158 @@
+package org.apache.lucene.search;
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+
+import org.apache.lucene.search.similarities.AfterEffect;
+import org.apache.lucene.search.similarities.AfterEffectB;
+import org.apache.lucene.search.similarities.AfterEffectL;
+import org.apache.lucene.search.similarities.BM25Similarity;
+import org.apache.lucene.search.similarities.BasicModel;
+import org.apache.lucene.search.similarities.BasicModelBE;
+import org.apache.lucene.search.similarities.BasicModelD;
+import org.apache.lucene.search.similarities.BasicModelG;
+import org.apache.lucene.search.similarities.BasicModelIF;
+import org.apache.lucene.search.similarities.BasicModelIn;
+import org.apache.lucene.search.similarities.BasicModelIne;
+import org.apache.lucene.search.similarities.BasicModelP;
+import org.apache.lucene.search.similarities.DFRSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Distribution;
+import org.apache.lucene.search.similarities.DistributionLL;
+import org.apache.lucene.search.similarities.DistributionSPL;
+import org.apache.lucene.search.similarities.IBSimilarity;
+import org.apache.lucene.search.similarities.LMDirichletSimilarity;
+import org.apache.lucene.search.similarities.LMJelinekMercerSimilarity;
+import org.apache.lucene.search.similarities.Lambda;
+import org.apache.lucene.search.similarities.LambdaDF;
+import org.apache.lucene.search.similarities.LambdaTTF;
+import org.apache.lucene.search.similarities.Normalization;
+import org.apache.lucene.search.similarities.NormalizationH1;
+import org.apache.lucene.search.similarities.NormalizationH2;
+import org.apache.lucene.search.similarities.NormalizationH3;
+import org.apache.lucene.search.similarities.NormalizationZ;
+import org.apache.lucene.search.similarities.Similarity;
+
+public class RandomSimilarityProvider extends DefaultSimilarityProvider {
+  final List<Similarity> knownSims;
+  Map<String,Similarity> previousMappings = new HashMap<String,Similarity>();
+  final int perFieldSeed;
+  final boolean shouldCoord;
+  final boolean shouldQueryNorm;
+  
+  public RandomSimilarityProvider(Random random) {
+    perFieldSeed = random.nextInt();
+    shouldCoord = random.nextBoolean();
+    shouldQueryNorm = random.nextBoolean();
+    knownSims = new ArrayList<Similarity>(allSims);
+    Collections.shuffle(knownSims, random);
+  }
+  
+  @Override
+  public float coord(int overlap, int maxOverlap) {
+    if (shouldCoord) {
+      return super.coord(overlap, maxOverlap);
+    } else {
+      return 1.0f;
+    }
+  }
+  
+  @Override
+  public float queryNorm(float sumOfSquaredWeights) {
+    if (shouldQueryNorm) {
+      return super.queryNorm(sumOfSquaredWeights);
+    } else {
+      return 1.0f;
+    }
+  }
+  
+  @Override
+  public synchronized Similarity get(String field) {
+    assert field != null;
+    Similarity sim = previousMappings.get(field);
+    if (sim == null) {
+      sim = knownSims.get(Math.abs(perFieldSeed ^ field.hashCode()) % knownSims.size());
+      previousMappings.put(field, sim);
+    }
+    return sim;
+  }
+  
+  // all the similarities that we rotate through
+  /** The DFR basic models to test. */
+  static BasicModel[] BASIC_MODELS = {
+    new BasicModelBE(), /* TODO: enable new BasicModelD(), */ new BasicModelG(),
+    new BasicModelIF(), new BasicModelIn(), new BasicModelIne(),
+    /* TODO: enable new BasicModelP() */
+  };
+  /** The DFR aftereffects to test. */
+  static AfterEffect[] AFTER_EFFECTS = {
+    new AfterEffectB(), new AfterEffectL(), new AfterEffect.NoAfterEffect()
+  };
+  /** The DFR normalizations to test. */
+  static Normalization[] NORMALIZATIONS = {
+    new NormalizationH1(), new NormalizationH2(),
+    new NormalizationH3(), new NormalizationZ()
+    // TODO: if we enable NoNormalization, we have to deal with
+    // a couple tests (e.g. TestDocBoost, TestSort) that expect length normalization
+    // new Normalization.NoNormalization()
+  };
+  /** The distributions for IB. */
+  static Distribution[] DISTRIBUTIONS = {
+    new DistributionLL(), new DistributionSPL()
+  };
+  /** Lambdas for IB. */
+  static Lambda[] LAMBDAS = {
+    new LambdaDF(), new LambdaTTF()
+  };
+  static List<Similarity> allSims;
+  static {
+    allSims = new ArrayList<Similarity>();
+    allSims.add(new DefaultSimilarity());
+    allSims.add(new BM25Similarity());
+    for (BasicModel basicModel : BASIC_MODELS) {
+      for (AfterEffect afterEffect : AFTER_EFFECTS) {
+        for (Normalization normalization : NORMALIZATIONS) {
+          allSims.add(new DFRSimilarity(basicModel, afterEffect, normalization));
+        }
+      }
+    }
+    for (Distribution distribution : DISTRIBUTIONS) {
+      for (Lambda lambda : LAMBDAS) {
+        for (Normalization normalization : NORMALIZATIONS) {
+          allSims.add(new IBSimilarity(distribution, lambda, normalization));
+        }
+      }
+    }
+    /* TODO: enable Dirichlet 
+    allSims.add(new LMDirichletSimilarity()); */
+    allSims.add(new LMJelinekMercerSimilarity(0.1f));
+    allSims.add(new LMJelinekMercerSimilarity(0.7f));
+  }
+  
+  @Override
+  public synchronized String toString() {
+    return "RandomSimilarityProvider(queryNorm=" + shouldQueryNorm + ",coord=" + shouldCoord + "): " + previousMappings.toString();
+  }
+}

Modified: lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/util/LuceneTestCase.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/util/LuceneTestCase.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/util/LuceneTestCase.java (original)
+++ lucene/dev/trunk/lucene/src/test-framework/org/apache/lucene/util/LuceneTestCase.java Sun Sep 11 15:47:21 2011
@@ -52,6 +52,8 @@ import org.apache.lucene.search.FieldCac
 import org.apache.lucene.search.FieldCache.CacheEntry;
 import org.apache.lucene.search.AssertingIndexSearcher;
 import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.RandomSimilarityProvider;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.FSDirectory;
 import org.apache.lucene.store.FlushInfo;
@@ -209,6 +211,8 @@ public abstract class LuceneTestCase ext
   private static Codec codec;
   // default codec provider
   private static CodecProvider savedCodecProvider;
+  
+  private static SimilarityProvider similarityProvider;
 
   private static Locale locale;
   private static Locale savedLocale;
@@ -393,6 +397,7 @@ public abstract class LuceneTestCase ext
     savedTimeZone = TimeZone.getDefault();
     timeZone = TEST_TIMEZONE.equals("random") ? randomTimeZone(random) : TimeZone.getTimeZone(TEST_TIMEZONE);
     TimeZone.setDefault(timeZone);
+    similarityProvider = new RandomSimilarityProvider(random);
     testsFailed = false;
   }
 
@@ -467,6 +472,7 @@ public abstract class LuceneTestCase ext
   /** print some useful debugging information about the environment */
   private static void printDebuggingInformation(String codecDescription) {
     System.err.println("NOTE: test params are: codec=" + codecDescription +
+        ", sim=" + similarityProvider +
         ", locale=" + locale +
         ", timezone=" + (timeZone == null ? "(null)" : timeZone.getID()));
     System.err.println("NOTE: all tests run in this JVM:");
@@ -922,6 +928,7 @@ public abstract class LuceneTestCase ext
   /** create a new index writer config with random defaults using the specified random */
   public static IndexWriterConfig newIndexWriterConfig(Random r, Version v, Analyzer a) {
     IndexWriterConfig c = new IndexWriterConfig(v, a);
+    c.setSimilarityProvider(similarityProvider);
     if (r.nextBoolean()) {
       c.setMergeScheduler(new SerialMergeScheduler());
     }
@@ -1249,7 +1256,9 @@ public abstract class LuceneTestCase ext
       if (maybeWrap && rarely()) {
         r = new SlowMultiReaderWrapper(r);
       }
-      return random.nextBoolean() ? new AssertingIndexSearcher(r) : new AssertingIndexSearcher(r.getTopReaderContext());
+      IndexSearcher ret = random.nextBoolean() ? new AssertingIndexSearcher(r) : new AssertingIndexSearcher(r.getTopReaderContext());
+      ret.setSimilarityProvider(similarityProvider);
+      return ret;
     } else {
       int threads = 0;
       final ExecutorService ex = (random.nextBoolean()) ? null
@@ -1258,7 +1267,7 @@ public abstract class LuceneTestCase ext
       if (ex != null && VERBOSE) {
         System.out.println("NOTE: newSearcher using ExecutorService with " + threads + " threads");
       }
-      return random.nextBoolean() ? 
+      IndexSearcher ret = random.nextBoolean() ? 
         new AssertingIndexSearcher(r, ex) {
           @Override
           public void close() throws IOException {
@@ -1272,6 +1281,8 @@ public abstract class LuceneTestCase ext
             shutdownExecutorService(ex);
           }
         };
+      ret.setSimilarityProvider(similarityProvider);
+      return ret;
     }
   }
   

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java Sun Sep 11 15:47:21 2011
@@ -35,13 +35,13 @@ import org.apache.lucene.document.String
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.FieldInfo.IndexOptions;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
-import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.DocIdSetIterator;
 import org.apache.lucene.search.FieldCache;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.NumericRangeQuery;
 import org.apache.lucene.search.ScoreDoc;
 import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
 import org.apache.lucene.store.CompoundFileDirectory;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.RAMDirectory;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestDeletionPolicy.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestDeletionPolicy.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestDeletionPolicy.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestDeletionPolicy.java Sun Sep 11 15:47:21 2011
@@ -27,11 +27,11 @@ import org.apache.lucene.analysis.MockAn
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
-import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.Query;
 import org.apache.lucene.search.ScoreDoc;
 import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexFileDeleter.java Sun Sep 11 15:47:21 2011
@@ -25,7 +25,7 @@ import org.apache.lucene.document.Docume
 import org.apache.lucene.document.StringField;
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
-import org.apache.lucene.search.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
 import org.apache.lucene.store.CompoundFileDirectory;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.IndexInput;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReader.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReader.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReader.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReader.java Sun Sep 11 15:47:21 2011
@@ -40,9 +40,9 @@ import org.apache.lucene.document.TextFi
 import org.apache.lucene.index.IndexReader.FieldOption;
 import org.apache.lucene.index.codecs.CodecProvider;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
-import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.DocIdSetIterator;
 import org.apache.lucene.search.FieldCache;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
 import org.apache.lucene.store.AlreadyClosedException;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.LockObtainFailedException;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderClone.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderClone.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderClone.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderClone.java Sun Sep 11 15:47:21 2011
@@ -17,7 +17,7 @@ package org.apache.lucene.index;
  * limitations under the License.
  */
 
-import org.apache.lucene.search.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
 import org.apache.lucene.analysis.MockAnalyzer;
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.TextField;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderCloneNorms.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderCloneNorms.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderCloneNorms.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderCloneNorms.java Sun Sep 11 15:47:21 2011
@@ -29,10 +29,10 @@ import org.apache.lucene.document.Field;
 import org.apache.lucene.document.FieldType;
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
-import org.apache.lucene.search.DefaultSimilarity;
-import org.apache.lucene.search.DefaultSimilarityProvider;
-import org.apache.lucene.search.Similarity;
-import org.apache.lucene.search.SimilarityProvider;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderOnDiskFull.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderOnDiskFull.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderOnDiskFull.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderOnDiskFull.java Sun Sep 11 15:47:21 2011
@@ -23,10 +23,11 @@ import org.apache.lucene.analysis.MockAn
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.StringField;
 import org.apache.lucene.document.TextField;
-import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.ScoreDoc;
 import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.Similarity;
 import org.apache.lucene.store.MockDirectoryWrapper;
 import org.apache.lucene.store.RAMDirectory;
 import org.apache.lucene.util.LuceneTestCase;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderReopen.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderReopen.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderReopen.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexReaderReopen.java Sun Sep 11 15:47:21 2011
@@ -35,11 +35,13 @@ import org.apache.lucene.document.FieldT
 import org.apache.lucene.document.StringField;
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
-import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.FieldCache;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.ScoreDoc;
 import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.AlreadyClosedException;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.BitVector;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexWriterConfig.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexWriterConfig.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexWriterConfig.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestIndexWriterConfig.java Sun Sep 11 15:47:21 2011
@@ -26,8 +26,8 @@ import java.util.Set;
 import org.apache.lucene.analysis.MockAnalyzer;
 import org.apache.lucene.index.DocumentsWriterPerThread.IndexingChain;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
-import org.apache.lucene.search.DefaultSimilarityProvider;
 import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
 import org.apache.lucene.util.LuceneTestCase;
 import org.junit.Test;
 

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestMaxTermFrequency.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestMaxTermFrequency.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestMaxTermFrequency.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestMaxTermFrequency.java Sun Sep 11 15:47:21 2011
@@ -27,9 +27,9 @@ import org.apache.lucene.analysis.MockTo
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Field;
 import org.apache.lucene.document.TextField;
-import org.apache.lucene.search.DefaultSimilarity;
-import org.apache.lucene.search.DefaultSimilarityProvider;
-import org.apache.lucene.search.Similarity;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 import org.apache.lucene.util._TestUtil;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestNorms.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestNorms.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestNorms.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestNorms.java Sun Sep 11 15:47:21 2011
@@ -27,10 +27,10 @@ import org.apache.lucene.document.Docume
 import org.apache.lucene.document.Field;
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
-import org.apache.lucene.search.DefaultSimilarity;
-import org.apache.lucene.search.DefaultSimilarityProvider;
-import org.apache.lucene.search.Similarity;
-import org.apache.lucene.search.SimilarityProvider;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestOmitTf.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestOmitTf.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestOmitTf.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestOmitTf.java Sun Sep 11 15:47:21 2011
@@ -32,6 +32,9 @@ import org.apache.lucene.document.TextFi
 import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.search.*;
 import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
+import org.apache.lucene.search.similarities.TFIDFSimilarity;
 import org.apache.lucene.store.Directory;
 
 

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestParallelReader.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestParallelReader.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestParallelReader.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestParallelReader.java Sun Sep 11 15:47:21 2011
@@ -25,6 +25,7 @@ import org.apache.lucene.analysis.MockAn
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
 import org.apache.lucene.search.*;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
@@ -234,7 +235,8 @@ public class TestParallelReader extends 
     w.addDocument(d2);
     w.close();
 
-    return new IndexSearcher(dir, false);
+    IndexReader ir = IndexReader.open(dir, false);
+    return newSearcher(ir);
   }
 
   // Fields 1 & 2 in one index, 3 & 4 in other, with ParallelReader:

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestUniqueTermCount.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestUniqueTermCount.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestUniqueTermCount.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/index/TestUniqueTermCount.java Sun Sep 11 15:47:21 2011
@@ -25,9 +25,9 @@ import org.apache.lucene.analysis.MockTo
 import org.apache.lucene.document.Document;
 import org.apache.lucene.document.Field;
 import org.apache.lucene.document.TextField;
-import org.apache.lucene.search.DefaultSimilarity;
-import org.apache.lucene.search.DefaultSimilarityProvider;
-import org.apache.lucene.search.Similarity;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 import org.apache.lucene.util._TestUtil;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/JustCompileSearch.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/JustCompileSearch.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/JustCompileSearch.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/JustCompileSearch.java Sun Sep 11 15:47:21 2011
@@ -20,9 +20,11 @@ package org.apache.lucene.search;
 import java.io.IOException;
 
 import org.apache.lucene.index.IndexReader.AtomicReaderContext;
-import org.apache.lucene.search.Similarity.ExactDocScorer;
-import org.apache.lucene.search.Similarity.SloppyDocScorer;
-import org.apache.lucene.search.Similarity.Stats;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity.ExactDocScorer;
+import org.apache.lucene.search.similarities.Similarity.SloppyDocScorer;
+import org.apache.lucene.search.similarities.Similarity.Stats;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.TermContext;
 import org.apache.lucene.index.FieldInvertState;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestBoolean2.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestBoolean2.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestBoolean2.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestBoolean2.java Sun Sep 11 15:47:21 2011
@@ -26,6 +26,8 @@ import org.apache.lucene.document.TextFi
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.IOContext;
 import org.apache.lucene.store.MockDirectoryWrapper;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestBooleanQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestBooleanQuery.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestBooleanQuery.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestBooleanQuery.java Sun Sep 11 15:47:21 2011
@@ -27,6 +27,9 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.index.MultiReader;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 import org.apache.lucene.util.NamedThreadFactory;
@@ -72,6 +75,21 @@ public class TestBooleanQuery extends Lu
 
     IndexReader r = w.getReader();
     IndexSearcher s = newSearcher(r);
+    // this test relies upon coord being the default implementation,
+    // otherwise scores are different!
+    final SimilarityProvider delegate = s.getSimilarityProvider();
+    s.setSimilarityProvider(new DefaultSimilarityProvider() {
+      @Override
+      public float queryNorm(float sumOfSquaredWeights) {
+        return delegate.queryNorm(sumOfSquaredWeights);
+      }
+
+      @Override
+      public Similarity get(String field) {
+        return delegate.get(field);
+      }
+    });
+
     BooleanQuery q = new BooleanQuery();
     q.add(new TermQuery(new Term("field", "a")), BooleanClause.Occur.SHOULD);
 
@@ -81,7 +99,7 @@ public class TestBooleanQuery extends Lu
     subQuery.setBoost(0);
     q.add(subQuery, BooleanClause.Occur.SHOULD);
     float score2 = s.search(q, 10).getMaxScore();
-    assertEquals(score*.5, score2, 1e-6);
+    assertEquals(score*.5F, score2, 1e-6);
 
     // LUCENE-2617: make sure that a clause not in the index still contributes to the score via coord factor
     BooleanQuery qq = (BooleanQuery)q.clone();
@@ -91,14 +109,14 @@ public class TestBooleanQuery extends Lu
     phrase.setBoost(0);
     qq.add(phrase, BooleanClause.Occur.SHOULD);
     score2 = s.search(qq, 10).getMaxScore();
-    assertEquals(score*(1.0/3), score2, 1e-6);
+    assertEquals(score*(1/3F), score2, 1e-6);
 
     // now test BooleanScorer2
     subQuery = new TermQuery(new Term("field", "b"));
     subQuery.setBoost(0);
     q.add(subQuery, BooleanClause.Occur.MUST);
     score2 = s.search(q, 10).getMaxScore();
-    assertEquals(score*(2.0/3), score2, 1e-6);
+    assertEquals(score*(2/3F), score2, 1e-6);
  
     // PhraseQuery w/ no terms added returns a null scorer
     PhraseQuery pq = new PhraseQuery();

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestComplexExplanations.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestComplexExplanations.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestComplexExplanations.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestComplexExplanations.java Sun Sep 11 15:47:21 2011
@@ -19,6 +19,7 @@ package org.apache.lucene.search;
 
 import org.apache.lucene.index.Term;
 import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
 import org.apache.lucene.search.spans.*;
 
 /**

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestConstantScoreQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestConstantScoreQuery.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestConstantScoreQuery.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestConstantScoreQuery.java Sun Sep 11 15:47:21 2011
@@ -23,6 +23,7 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDisjunctionMaxQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDisjunctionMaxQuery.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDisjunctionMaxQuery.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDisjunctionMaxQuery.java Sun Sep 11 15:47:21 2011
@@ -29,6 +29,10 @@ import org.apache.lucene.index.FieldInve
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.search.Weight.ScorerContext;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 
 import java.text.DecimalFormat;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDocBoost.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDocBoost.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDocBoost.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDocBoost.java Sun Sep 11 15:47:21 2011
@@ -56,7 +56,8 @@ public class TestDocBoost extends Lucene
 
     final float[] scores = new float[4];
 
-    newSearcher(reader).search
+    IndexSearcher searcher = newSearcher(reader);
+    searcher.search
       (new TermQuery(new Term("field", "word")),
        new Collector() {
          private int base = 0;
@@ -82,7 +83,10 @@ public class TestDocBoost extends Lucene
     float lastScore = 0.0f;
 
     for (int i = 0; i < 2; i++) {
-      assertTrue(scores[i] > lastScore);
+      if (VERBOSE) {
+        System.out.println(searcher.explain(new TermQuery(new Term("field", "word")), i));
+      }
+      assertTrue("score: " + scores[i] + " should be > lastScore: " + lastScore, scores[i] > lastScore);
       lastScore = scores[i];
     }
     

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDocValuesScoring.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDocValuesScoring.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDocValuesScoring.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestDocValuesScoring.java Sun Sep 11 15:47:21 2011
@@ -30,6 +30,9 @@ import org.apache.lucene.index.Term;
 import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.codecs.CodecProvider;
 import org.apache.lucene.index.values.IndexDocValues.Source;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.LuceneTestCase;
@@ -71,13 +74,24 @@ public class TestDocValuesScoring extend
     
     // no boosting
     IndexSearcher searcher1 = newSearcher(ir);
+    final SimilarityProvider base = searcher1.getSimilarityProvider();
     // boosting
     IndexSearcher searcher2 = newSearcher(ir);
-    searcher2.setSimilarityProvider(new DefaultSimilarityProvider() {
-      final Similarity fooSim = new BoostingSimilarity(super.get("foo"), "foo_boost");
+    searcher2.setSimilarityProvider(new SimilarityProvider() {
+      final Similarity fooSim = new BoostingSimilarity(base.get("foo"), "foo_boost");
 
       public Similarity get(String field) {
-        return "foo".equals(field) ? fooSim : super.get(field);
+        return "foo".equals(field) ? fooSim : base.get(field);
+      }
+
+      @Override
+      public float coord(int overlap, int maxOverlap) {
+        return base.coord(overlap, maxOverlap);
+      }
+
+      @Override
+      public float queryNorm(float sumOfSquaredWeights) {
+        return base.queryNorm(sumOfSquaredWeights);
       }
     });
     

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestElevationComparator.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestElevationComparator.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestElevationComparator.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestElevationComparator.java Sun Sep 11 15:47:21 2011
@@ -23,6 +23,7 @@ import org.apache.lucene.document.TextFi
 import org.apache.lucene.index.*;
 import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.search.FieldValueHitQueue.Entry;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
 import org.apache.lucene.store.*;
 import org.apache.lucene.util.LuceneTestCase;
 import org.apache.lucene.util.BytesRef;
@@ -41,7 +42,8 @@ public class TestElevationComparator ext
         directory,
         newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random)).
             setMaxBufferedDocs(2).
-            setMergePolicy(newLogMergePolicy(1000))
+            setMergePolicy(newLogMergePolicy(1000)).
+            setSimilarityProvider(new DefaultSimilarityProvider())
     );
     writer.addDocument(adoc(new String[] {"id", "a", "title", "ipod", "str_s", "a"}));
     writer.addDocument(adoc(new String[] {"id", "b", "title", "ipod ipod", "str_s", "b"}));
@@ -54,6 +56,7 @@ public class TestElevationComparator ext
     writer.close();
 
     IndexSearcher searcher = newSearcher(r);
+    searcher.setSimilarityProvider(new DefaultSimilarityProvider());
 
     runTest(searcher, true);
     runTest(searcher, false);

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestFuzzyQuery2.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestFuzzyQuery2.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestFuzzyQuery2.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestFuzzyQuery2.java Sun Sep 11 15:47:21 2011
@@ -29,6 +29,9 @@ import org.apache.lucene.document.TextFi
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 
@@ -104,6 +107,21 @@ public class TestFuzzyQuery2 extends Luc
     if (VERBOSE) {
       System.out.println("TEST: searcher=" + searcher);
     }
+    // even though this uses a boost-only rewrite, this test relies upon queryNorm being the default implementation,
+    // otherwise scores are different!
+    final SimilarityProvider delegate = searcher.getSimilarityProvider();
+    searcher.setSimilarityProvider(new DefaultSimilarityProvider() {
+      @Override
+      public float coord(int overlap, int maxOverlap) {
+        return delegate.coord(overlap, maxOverlap);
+      }
+
+      @Override
+      public Similarity get(String field) {
+        return delegate.get(field);
+      }
+    });
+    
     writer.close();
     String line;
     while ((line = reader.readLine()) != null) {

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestMultiPhraseQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestMultiPhraseQuery.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestMultiPhraseQuery.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestMultiPhraseQuery.java Sun Sep 11 15:47:21 2011
@@ -37,6 +37,9 @@ import org.apache.lucene.document.String
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
 import org.apache.lucene.store.RAMDirectory;
 import org.apache.lucene.util.LuceneTestCase;
 

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestMultiTermConstantScore.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestMultiTermConstantScore.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestMultiTermConstantScore.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestMultiTermConstantScore.java Sun Sep 11 15:47:21 2011
@@ -26,6 +26,9 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.junit.AfterClass;
 import org.junit.BeforeClass;
@@ -169,6 +172,19 @@ public class TestMultiTermConstantScore 
 
     // test for correct application of query normalization
     // must use a non score normalizing method for this.
+    
+    final SimilarityProvider delegate = search.getSimilarityProvider();
+    search.setSimilarityProvider(new DefaultSimilarityProvider() {
+      @Override
+      public float coord(int overlap, int maxOverlap) {
+        return delegate.coord(overlap, maxOverlap);
+      }
+
+      @Override
+      public Similarity get(String field) {
+        return delegate.get(field);
+      }
+    });
     Query q = csrq("data", "1", "6", T, T);
     q.setBoost(100);
     search.search(q, null, new Collector() {

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestPhraseQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestPhraseQuery.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestPhraseQuery.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestPhraseQuery.java Sun Sep 11 15:47:21 2011
@@ -23,6 +23,7 @@ import org.apache.lucene.analysis.tokena
 import org.apache.lucene.document.*;
 import org.apache.lucene.index.*;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
 import org.apache.lucene.store.*;
 import org.apache.lucene.util.Version;
 import org.apache.lucene.util._TestUtil;
@@ -342,7 +343,10 @@ public class TestPhraseQuery extends Luc
   
   public void testSlopScoring() throws IOException {
     Directory directory = newDirectory();
-    RandomIndexWriter writer = new RandomIndexWriter(random, directory, newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random)).setMergePolicy(newLogMergePolicy()));
+    RandomIndexWriter writer = new RandomIndexWriter(random, directory, 
+        newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random))
+          .setMergePolicy(newLogMergePolicy())
+          .setSimilarityProvider(new DefaultSimilarityProvider()));
 
     Document doc = new Document();
     doc.add(newField("field", "foo firstname lastname foo", TextField.TYPE_STORED));
@@ -360,6 +364,7 @@ public class TestPhraseQuery extends Luc
     writer.close();
 
     IndexSearcher searcher = newSearcher(reader);
+    searcher.setSimilarityProvider(new DefaultSimilarityProvider());
     PhraseQuery query = new PhraseQuery();
     query.add(new Term("field", "firstname"));
     query.add(new Term("field", "lastname"));

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSetNorm.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSetNorm.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSetNorm.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSetNorm.java Sun Sep 11 15:47:21 2011
@@ -26,6 +26,7 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.IndexWriter;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
 import org.apache.lucene.store.Directory;
 
 /** Document boost unit test.

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSimilarity.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSimilarity.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSimilarity.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSimilarity.java Sun Sep 11 15:47:21 2011
@@ -27,6 +27,9 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.analysis.MockAnalyzer;
 import org.apache.lucene.document.Document;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSimilarityProvider.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSimilarityProvider.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSimilarityProvider.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSimilarityProvider.java Sun Sep 11 15:47:21 2011
@@ -27,6 +27,9 @@ import org.apache.lucene.index.IndexWrit
 import org.apache.lucene.index.MultiNorms;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
+import org.apache.lucene.search.similarities.TFIDFSimilarity;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.LuceneTestCase;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSloppyPhraseQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSloppyPhraseQuery.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSloppyPhraseQuery.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestSloppyPhraseQuery.java Sun Sep 11 15:47:21 2011
@@ -17,6 +17,8 @@ package org.apache.lucene.search;
  * limitations under the License.
  */
 
+import java.io.IOException;
+
 import org.apache.lucene.util.LuceneTestCase;
 import org.apache.lucene.analysis.MockAnalyzer;
 import org.apache.lucene.analysis.MockTokenizer;
@@ -25,6 +27,7 @@ import org.apache.lucene.document.Field;
 import org.apache.lucene.document.FieldType;
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexReader.AtomicReaderContext;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.store.Directory;
@@ -68,9 +71,9 @@ public class TestSloppyPhraseQuery exten
    */
   public void testDoc1_Query1_All_Slops_Should_match() throws Exception {
     for (int slop=0; slop<30; slop++) {
-      float score1 = checkPhraseQuery(DOC_1, QUERY_1, slop, 1);
-      float score2 = checkPhraseQuery(DOC_1_B, QUERY_1, slop, 1);
-      assertTrue("slop="+slop+" score2="+score2+" should be greater than score1 "+score1, score2>score1);
+      float freq1 = checkPhraseQuery(DOC_1, QUERY_1, slop, 1);
+      float freq2 = checkPhraseQuery(DOC_1_B, QUERY_1, slop, 1);
+      assertTrue("slop="+slop+" freq2="+freq2+" should be greater than score1 "+freq1, freq2>freq1);
     }
   }
 
@@ -82,10 +85,10 @@ public class TestSloppyPhraseQuery exten
   public void testDoc2_Query1_Slop_6_or_more_Should_match() throws Exception {
     for (int slop=0; slop<30; slop++) {
       int numResultsExpected = slop<6 ? 0 : 1;
-      float score1 = checkPhraseQuery(DOC_2, QUERY_1, slop, numResultsExpected);
+      float freq1 = checkPhraseQuery(DOC_2, QUERY_1, slop, numResultsExpected);
       if (numResultsExpected>0) {
-        float score2 = checkPhraseQuery(DOC_2_B, QUERY_1, slop, 1);
-        assertTrue("slop="+slop+" score2="+score2+" should be greater than score1 "+score1, score2>score1);
+        float freq2 = checkPhraseQuery(DOC_2_B, QUERY_1, slop, 1);
+        assertTrue("slop="+slop+" freq2="+freq2+" should be greater than freq1 "+freq1, freq2>freq1);
       }
     }
   }
@@ -97,9 +100,9 @@ public class TestSloppyPhraseQuery exten
    */
   public void testDoc2_Query2_All_Slops_Should_match() throws Exception {
     for (int slop=0; slop<30; slop++) {
-      float score1 = checkPhraseQuery(DOC_2, QUERY_2, slop, 1);
-      float score2 = checkPhraseQuery(DOC_2_B, QUERY_2, slop, 1);
-      assertTrue("slop="+slop+" score2="+score2+" should be greater than score1 "+score1, score2>score1);
+      float freq1 = checkPhraseQuery(DOC_2, QUERY_2, slop, 1);
+      float freq2 = checkPhraseQuery(DOC_2_B, QUERY_2, slop, 1);
+      assertTrue("slop="+slop+" freq2="+freq2+" should be greater than freq1 "+freq1, freq2>freq1);
     }
   }
 
@@ -109,9 +112,9 @@ public class TestSloppyPhraseQuery exten
    */
   public void testDoc3_Query1_All_Slops_Should_match() throws Exception {
     for (int slop=0; slop<30; slop++) {
-      float score1 = checkPhraseQuery(DOC_3, QUERY_1, slop, 1);
-      float score2 = checkPhraseQuery(DOC_3_B, QUERY_1, slop, 1);
-      assertTrue("slop="+slop+" score2="+score2+" should be greater than score1 "+score1, score2>score1);
+      float freq1 = checkPhraseQuery(DOC_3, QUERY_1, slop, 1);
+      float freq2 = checkPhraseQuery(DOC_3_B, QUERY_1, slop, 1);
+      assertTrue("slop="+slop+" freq2="+freq2+" should be greater than freq1 "+freq1, freq2>freq1);
     }
   }
 
@@ -140,9 +143,9 @@ public class TestSloppyPhraseQuery exten
     IndexReader reader = writer.getReader();
 
     IndexSearcher searcher = newSearcher(reader);
-    TopDocs td = searcher.search(query,null,10);
-    //System.out.println("slop: "+slop+"  query: "+query+"  doc: "+doc+"  Expecting number of hits: "+expectedNumResults+" maxScore="+td.getMaxScore());
-    assertEquals("slop: "+slop+"  query: "+query+"  doc: "+doc+"  Wrong number of hits", expectedNumResults, td.totalHits);
+    MaxFreqCollector c = new MaxFreqCollector();
+    searcher.search(query, c);
+    assertEquals("slop: "+slop+"  query: "+query+"  doc: "+doc+"  Wrong number of hits", expectedNumResults, c.totalHits);
 
     //QueryUtils.check(query,searcher);
     writer.close();
@@ -150,7 +153,9 @@ public class TestSloppyPhraseQuery exten
     reader.close();
     ramDir.close();
 
-    return td.getMaxScore();
+    // returns the max Scorer.freq() found, because even though norms are omitted, many index stats are different
+    // with these different tokens/distributions/lengths.. otherwise this test is very fragile.
+    return c.max; 
   }
 
   private static Document makeDocument(String docText) {
@@ -171,4 +176,29 @@ public class TestSloppyPhraseQuery exten
     return query;
   }
 
+  static class MaxFreqCollector extends Collector {
+    float max;
+    int totalHits;
+    Scorer scorer;
+    
+    @Override
+    public void setScorer(Scorer scorer) throws IOException {
+      this.scorer = scorer;
+    }
+
+    @Override
+    public void collect(int doc) throws IOException {
+      totalHits++;
+      max = Math.max(max, scorer.freq());
+    }
+
+    @Override
+    public void setNextReader(AtomicReaderContext context) throws IOException {      
+    }
+
+    @Override
+    public boolean acceptsDocsOutOfOrder() {
+      return false;
+    }
+  }
 }

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestTermScorer.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestTermScorer.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestTermScorer.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestTermScorer.java Sun Sep 11 15:47:21 2011
@@ -30,6 +30,7 @@ import org.apache.lucene.index.RandomInd
 import org.apache.lucene.index.SlowMultiReaderWrapper;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.search.Weight.ScorerContext;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.LuceneTestCase;
 
@@ -47,7 +48,10 @@ public class TestTermScorer extends Luce
     super.setUp();
     directory = newDirectory();
     
-    RandomIndexWriter writer = new RandomIndexWriter(random, directory, newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random)).setMergePolicy(newLogMergePolicy()));
+    RandomIndexWriter writer = new RandomIndexWriter(random, directory, 
+        newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random))
+        .setMergePolicy(newLogMergePolicy())
+        .setSimilarityProvider(new DefaultSimilarityProvider()));
     for (int i = 0; i < values.length; i++) {
       Document doc = new Document();
       doc
@@ -57,6 +61,7 @@ public class TestTermScorer extends Luce
     indexReader = new SlowMultiReaderWrapper(writer.getReader());
     writer.close();
     indexSearcher = newSearcher(indexReader);
+    indexSearcher.setSimilarityProvider(new DefaultSimilarityProvider());
   }
   
   @Override

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestTermVectors.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestTermVectors.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestTermVectors.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/TestTermVectors.java Sun Sep 11 15:47:21 2011
@@ -28,6 +28,7 @@ import org.apache.lucene.document.FieldT
 import org.apache.lucene.document.TextField;
 import org.apache.lucene.index.*;
 import org.apache.lucene.index.IndexWriterConfig.OpenMode;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.util.English;
 
@@ -244,7 +245,9 @@ public class TestTermVectors extends Luc
     
     RandomIndexWriter writer = new RandomIndexWriter(random, dir, 
         newIndexWriterConfig(TEST_VERSION_CURRENT, new MockAnalyzer(random, MockTokenizer.SIMPLE, true))
-                                                     .setOpenMode(OpenMode.CREATE).setMergePolicy(newLogMergePolicy()));
+          .setOpenMode(OpenMode.CREATE)
+          .setMergePolicy(newLogMergePolicy())
+          .setSimilarityProvider(new DefaultSimilarityProvider()));
     writer.addDocument(testDoc1);
     writer.addDocument(testDoc2);
     writer.addDocument(testDoc3);
@@ -252,6 +255,7 @@ public class TestTermVectors extends Luc
     IndexReader reader = writer.getReader();
     writer.close();
     IndexSearcher knownSearcher = newSearcher(reader);
+    knownSearcher.setSimilarityProvider(new DefaultSimilarityProvider());
     FieldsEnum fields = MultiFields.getFields(knownSearcher.reader).iterator();
     
     DocsEnum docs = null;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/PayloadHelper.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/PayloadHelper.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/PayloadHelper.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/PayloadHelper.java Sun Sep 11 15:47:21 2011
@@ -29,7 +29,7 @@ import org.apache.lucene.util.English;
 import org.apache.lucene.util.LuceneTestCase;
 import org.apache.lucene.index.IndexReader;
 import org.apache.lucene.search.IndexSearcher;
-import org.apache.lucene.search.SimilarityProvider;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.store.Directory;
 import org.apache.lucene.store.MockDirectoryWrapper;
 import org.apache.lucene.store.RAMDirectory;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadExplanations.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadExplanations.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadExplanations.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadExplanations.java Sun Sep 11 15:47:21 2011
@@ -18,9 +18,9 @@ package org.apache.lucene.search.payload
  */
 
 import org.apache.lucene.index.Term;
-import org.apache.lucene.search.DefaultSimilarity;
-import org.apache.lucene.search.DefaultSimilarityProvider;
-import org.apache.lucene.search.Similarity;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
 import org.apache.lucene.search.TestExplanations;
 import org.apache.lucene.search.spans.SpanQuery;
 import org.apache.lucene.util.BytesRef;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadNearQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadNearQuery.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadNearQuery.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadNearQuery.java Sun Sep 11 15:47:21 2011
@@ -27,14 +27,14 @@ import org.apache.lucene.index.IndexRead
 import org.apache.lucene.index.Payload;
 import org.apache.lucene.index.RandomIndexWriter;
 import org.apache.lucene.index.Term;
-import org.apache.lucene.search.DefaultSimilarity;
 import org.apache.lucene.search.Explanation;
 import org.apache.lucene.search.QueryUtils;
 import org.apache.lucene.search.ScoreDoc;
 import org.apache.lucene.search.IndexSearcher;
-import org.apache.lucene.search.Similarity;
-import org.apache.lucene.search.SimilarityProvider;
 import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.search.spans.SpanQuery;
 import org.apache.lucene.search.spans.SpanNearQuery;
 import org.apache.lucene.search.spans.SpanTermQuery;

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadTermQuery.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadTermQuery.java?rev=1169470&r1=1169469&r2=1169470&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadTermQuery.java (original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/payloads/TestPayloadTermQuery.java Sun Sep 11 15:47:21 2011
@@ -20,17 +20,17 @@ import org.apache.lucene.analysis.*;
 import org.apache.lucene.util.BytesRef;
 import org.apache.lucene.util.LuceneTestCase;
 import org.apache.lucene.util.English;
-import org.apache.lucene.search.DefaultSimilarityProvider;
 import org.apache.lucene.search.IndexSearcher;
 import org.apache.lucene.search.QueryUtils;
-import org.apache.lucene.search.Similarity;
-import org.apache.lucene.search.SimilarityProvider;
 import org.apache.lucene.search.TopDocs;
 import org.apache.lucene.search.ScoreDoc;
 import org.apache.lucene.search.CheckHits;
 import org.apache.lucene.search.BooleanClause;
 import org.apache.lucene.search.BooleanQuery;
-import org.apache.lucene.search.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarity;
+import org.apache.lucene.search.similarities.DefaultSimilarityProvider;
+import org.apache.lucene.search.similarities.Similarity;
+import org.apache.lucene.search.similarities.SimilarityProvider;
 import org.apache.lucene.search.spans.MultiSpansWrapper;
 import org.apache.lucene.search.spans.SpanTermQuery;
 import org.apache.lucene.search.spans.Spans;

Added: lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/similarities/SpoofIndexSearcher.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/similarities/SpoofIndexSearcher.java?rev=1169470&view=auto
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/similarities/SpoofIndexSearcher.java (added)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/search/similarities/SpoofIndexSearcher.java Sun Sep 11 15:47:21 2011
@@ -0,0 +1,216 @@
+package org.apache.lucene.search.similarities;
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Comparator;
+import java.util.Map;
+
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.Fields;
+import org.apache.lucene.index.FieldsEnum;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.StoredFieldVisitor;
+import org.apache.lucene.index.TermFreqVector;
+import org.apache.lucene.index.TermVectorMapper;
+import org.apache.lucene.index.Terms;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.index.codecs.PerDocValues;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.util.Bits;
+import org.apache.lucene.util.BytesRef;
+
+/**
+ * Index searcher implementation that takes an {@link BasicStats} instance and
+ * returns statistics accordingly. Most of the methods are not implemented, so
+ * it can only be used for Similarity unit testing.
+ */
+public class SpoofIndexSearcher extends IndexSearcher {
+  public SpoofIndexSearcher(BasicStats stats) {
+    super(new SpoofIndexReader(stats));
+  }
+  
+  public static class SpoofIndexReader extends IndexReader {
+    /** The stats the reader has to return. */
+    protected BasicStats stats;
+    /** The fields the reader has to return. */
+    protected SpoofFields fields;
+    
+    public SpoofIndexReader(BasicStats stats) {
+      this.stats = stats;
+      this.fields = new SpoofFields(stats);
+    }
+
+    @Override
+    public int numDocs() {
+      return stats.getNumberOfDocuments();
+    }
+
+    @Override
+    public int maxDoc() {
+      return stats.getNumberOfDocuments();
+    }
+
+    @Override
+    public Fields fields() throws IOException {
+      return fields;
+    }
+
+    @Override
+    public Collection<String> getFieldNames(FieldOption fldOption) {
+      return Arrays.asList(new String[]{"spoof"});
+    }
+
+    @Override
+    public ReaderContext getTopReaderContext() {
+      return new AtomicReaderContext(this);
+    }
+    
+    @Override
+    public boolean hasDeletions() {
+      return false;
+    }
+
+    // ------------------------ Not implemented methods ------------------------
+    
+    @Override
+    public TermFreqVector[] getTermFreqVectors(int docNumber)
+        throws IOException {
+      return null;
+    }
+
+    @Override
+    public TermFreqVector getTermFreqVector(int docNumber, String field)
+        throws IOException {
+      return null;
+    }
+
+    @Override
+    public void getTermFreqVector(int docNumber, String field,
+        TermVectorMapper mapper) throws IOException {
+    }
+
+    @Override
+    public void getTermFreqVector(int docNumber, TermVectorMapper mapper)
+        throws IOException {
+    }
+
+    @Override
+    public void document(int docID, StoredFieldVisitor visitor) throws CorruptIndexException, IOException {
+    }
+
+    @Override
+    public byte[] norms(String field) throws IOException {
+      return null;
+    }
+
+    @Override
+    protected void doSetNorm(int doc, String field, byte value)
+        throws CorruptIndexException, IOException {
+    }
+
+    @Override
+    public PerDocValues perDocValues() throws IOException {
+      return null;
+    }
+
+    @Override
+    protected void doDelete(int docNum) throws CorruptIndexException,
+        IOException {
+    }
+
+    @Override
+    protected void doUndeleteAll() throws CorruptIndexException, IOException {
+    }
+
+    @Override
+    protected void doCommit(Map<String,String> commitUserData)
+        throws IOException {
+    }
+
+    @Override
+    protected void doClose() throws IOException {
+    }
+
+    @Override
+    public Bits getLiveDocs() {
+      return null;
+    }
+  }
+  
+  /** Spoof Fields class for Similarity testing. */
+  public static class SpoofFields extends Fields {
+    /** The stats the object has to return. */
+    protected SpoofTerms terms;
+    
+    public SpoofFields(BasicStats stats) {
+      this.terms = new SpoofTerms(stats);
+    }
+    
+    @Override
+    public Terms terms(String field) throws IOException {
+      return terms;
+    }
+    
+    // ------------------------ Not implemented methods ------------------------
+
+    @Override
+    public FieldsEnum iterator() throws IOException {
+      return null;
+    }
+  }
+  
+  /** Spoof Terms class for Similarity testing. */
+  public static class SpoofTerms extends Terms {
+    /** The stats the object has to return. */
+    protected BasicStats stats;
+    
+    public SpoofTerms(BasicStats stats) {
+      this.stats = stats;
+    }
+    
+    @Override
+    public long getSumTotalTermFreq() throws IOException {
+      return stats.getNumberOfFieldTokens();
+    }
+
+    @Override
+    public long getSumDocFreq() throws IOException {
+      return stats.getDocFreq();
+    }
+    
+    @Override
+    public int getDocCount() throws IOException {
+      return stats.getDocFreq();
+    }    
+    
+    // ------------------------ Not implemented methods ------------------------
+
+    @Override
+    public TermsEnum iterator() throws IOException {
+      return null;
+    }
+
+    @Override
+    public Comparator<BytesRef> getComparator() throws IOException {
+      return null;
+    }
+  }
+}



Mime
View raw message