Return-Path: X-Original-To: apmail-lucene-commits-archive@www.apache.org Delivered-To: apmail-lucene-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AC12110143 for ; Sat, 29 Mar 2014 02:39:39 +0000 (UTC) Received: (qmail 653 invoked by uid 500); 29 Mar 2014 02:39:39 -0000 Mailing-List: contact commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list commits@lucene.apache.org Received: (qmail 645 invoked by uid 99); 29 Mar 2014 02:39:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Mar 2014 02:39:39 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Mar 2014 02:39:35 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id 9E30623889CB; Sat, 29 Mar 2014 02:39:15 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1582937 - in /lucene/dev/branches/lucene_solr_4_7: ./ dev-tools/ dev-tools/idea/solr/core/src/test/ solr/ solr/contrib/ solr/contrib/analysis-extras/src/java/org/apache/solr/schema/ solr/core/ solr/core/src/java/org/apache/solr/schema/ sol... Date: Sat, 29 Mar 2014 02:39:14 -0000 To: commits@lucene.apache.org From: sarowe@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20140329023915.9E30623889CB@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: sarowe Date: Sat Mar 29 02:39:14 2014 New Revision: 1582937 URL: http://svn.apache.org/r1582937 Log: SOLR-5920: Distributed sort on DateField, BoolField and BCD{Int,Long,Str}Field returns string cast exception (merged branch_4x r1582933) Modified: lucene/dev/branches/lucene_solr_4_7/ (props changed) lucene/dev/branches/lucene_solr_4_7/dev-tools/ (props changed) lucene/dev/branches/lucene_solr_4_7/dev-tools/idea/solr/core/src/test/solr-core-tests.iml lucene/dev/branches/lucene_solr_4_7/solr/ (props changed) lucene/dev/branches/lucene_solr_4_7/solr/CHANGES.txt (contents, props changed) lucene/dev/branches/lucene_solr_4_7/solr/common-build.xml (contents, props changed) lucene/dev/branches/lucene_solr_4_7/solr/contrib/ (props changed) lucene/dev/branches/lucene_solr_4_7/solr/contrib/analysis-extras/src/java/org/apache/solr/schema/ICUCollationField.java lucene/dev/branches/lucene_solr_4_7/solr/core/ (props changed) lucene/dev/branches/lucene_solr_4_7/solr/core/build.xml lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BCDIntField.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BoolField.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/CollationField.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/DateField.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/FieldType.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/StrField.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TextField.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TrieDateField.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/test-files/solr/collection1/conf/schema-sorts.xml lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/CursorPagingTest.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/schema/SortableBinaryField.java lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/search/CursorMarkTest.java Modified: lucene/dev/branches/lucene_solr_4_7/dev-tools/idea/solr/core/src/test/solr-core-tests.iml URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/dev-tools/idea/solr/core/src/test/solr-core-tests.iml?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/dev-tools/idea/solr/core/src/test/solr-core-tests.iml (original) +++ lucene/dev/branches/lucene_solr_4_7/dev-tools/idea/solr/core/src/test/solr-core-tests.iml Sat Mar 29 02:39:14 2014 @@ -15,6 +15,7 @@ + @@ -29,5 +30,7 @@ + + Modified: lucene/dev/branches/lucene_solr_4_7/solr/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/CHANGES.txt?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/CHANGES.txt (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/CHANGES.txt Sat Mar 29 02:39:14 2014 @@ -121,6 +121,9 @@ Bug Fixes * SOLR-5906: Collection create API ignores property.instanceDir parameter. (Varun Thacker, shalin) +* SOLR-5920: Distributed sort on DateField, BoolField and BCD{Int,Long,Str}Field + returns string cast exception (Eric Bus, AJ Lemke, hossman, Steve Rowe) + Other Changes --------------------- Modified: lucene/dev/branches/lucene_solr_4_7/solr/common-build.xml URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/common-build.xml?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/common-build.xml (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/common-build.xml Sat Mar 29 02:39:14 2014 @@ -454,6 +454,12 @@ + + + + + + Modified: lucene/dev/branches/lucene_solr_4_7/solr/contrib/analysis-extras/src/java/org/apache/solr/schema/ICUCollationField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/contrib/analysis-extras/src/java/org/apache/solr/schema/ICUCollationField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/contrib/analysis-extras/src/java/org/apache/solr/schema/ICUCollationField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/contrib/analysis-extras/src/java/org/apache/solr/schema/ICUCollationField.java Sat Mar 29 02:39:14 2014 @@ -43,7 +43,6 @@ import org.apache.lucene.util.Version; import org.apache.lucene.analysis.util.ResourceLoader; import org.apache.solr.common.SolrException; import org.apache.solr.common.SolrException.ErrorCode; -import org.apache.solr.common.util.Base64; import org.apache.solr.response.TextResponseWriter; import org.apache.solr.search.QParser; @@ -307,20 +306,11 @@ public class ICUCollationField extends F @Override public Object marshalSortValue(Object value) { - if (null == value) { - return null; - } - final BytesRef val = (BytesRef)value; - return Base64.byteArrayToBase64(val.bytes, val.offset, val.length); + return marshalBase64SortValue(value); } @Override public Object unmarshalSortValue(Object value) { - if (null == value) { - return null; - } - final String val = (String)value; - final byte[] bytes = Base64.base64ToByteArray(val); - return new BytesRef(bytes); + return unmarshalBase64SortValue(value); } } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/build.xml URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/build.xml?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/build.xml (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/build.xml Sat Mar 29 02:39:14 2014 @@ -32,9 +32,14 @@ + + + + + Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BCDIntField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BCDIntField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BCDIntField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BCDIntField.java Sat Mar 29 02:39:14 2014 @@ -65,6 +65,16 @@ public class BCDIntField extends Primiti public void write(TextResponseWriter writer, String name, IndexableField f) throws IOException { writer.writeInt(name,toExternal(f)); } + + @Override + public Object marshalSortValue(Object value) { + return marshalStringSortValue(value); + } + + @Override + public Object unmarshalSortValue(Object value) { + return unmarshalStringSortValue(value); + } } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BoolField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BoolField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BoolField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/BoolField.java Sat Mar 29 02:39:14 2014 @@ -150,6 +150,16 @@ public class BoolField extends Primitive public void write(TextResponseWriter writer, String name, IndexableField f) throws IOException { writer.writeBool(name, f.stringValue().charAt(0) == 'T'); } + + @Override + public Object marshalSortValue(Object value) { + return marshalStringSortValue(value); + } + + @Override + public Object unmarshalSortValue(Object value) { + return unmarshalStringSortValue(value); + } } // TODO - this can be much more efficient - use OpenBitSet or Bits Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/CollationField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/CollationField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/CollationField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/CollationField.java Sat Mar 29 02:39:14 2014 @@ -47,7 +47,6 @@ import org.apache.lucene.util.Version; import org.apache.lucene.analysis.util.ResourceLoader; import org.apache.solr.common.SolrException; import org.apache.solr.common.SolrException.ErrorCode; -import org.apache.solr.common.util.Base64; import org.apache.solr.response.TextResponseWriter; import org.apache.solr.search.QParser; @@ -282,20 +281,11 @@ public class CollationField extends Fiel @Override public Object marshalSortValue(Object value) { - if (null == value) { - return null; - } - final BytesRef val = (BytesRef)value; - return Base64.byteArrayToBase64(val.bytes, val.offset, val.length); + return marshalBase64SortValue(value); } @Override public Object unmarshalSortValue(Object value) { - if (null == value) { - return null; - } - final String val = (String)value; - final byte[] bytes = Base64.base64ToByteArray(val); - return new BytesRef(bytes); + return unmarshalBase64SortValue(value); } } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/DateField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/DateField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/DateField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/DateField.java Sat Mar 29 02:39:14 2014 @@ -245,6 +245,16 @@ public class DateField extends Primitive } @Override + public Object marshalSortValue(Object value) { + return marshalStringSortValue(value); + } + + @Override + public Object unmarshalSortValue(Object value) { + return unmarshalStringSortValue(value); + } + + @Override public void write(TextResponseWriter writer, String name, IndexableField f) throws IOException { writer.writeDate(name, toExternal(f)); } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/FieldType.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/FieldType.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/FieldType.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/FieldType.java Sat Mar 29 02:39:14 2014 @@ -48,6 +48,7 @@ import org.apache.solr.analysis.SolrAnal import org.apache.solr.analysis.TokenizerChain; import org.apache.solr.common.SolrException; import org.apache.solr.common.SolrException.ErrorCode; +import org.apache.solr.common.util.Base64; import org.apache.solr.common.util.SimpleOrderedMap; import org.apache.solr.common.util.StrUtils; import org.apache.solr.response.TextResponseWriter; @@ -968,4 +969,52 @@ public abstract class FieldType extends public Object unmarshalSortValue(Object value) { return value; } + + /** + * Marshals a string-based field value. + */ + protected static Object marshalStringSortValue(Object value) { + if (null == value) { + return null; + } + CharsRef spare = new CharsRef(); + UnicodeUtil.UTF8toUTF16((BytesRef)value, spare); + return spare.toString(); + } + + /** + * Unmarshals a string-based field value. + */ + protected static Object unmarshalStringSortValue(Object value) { + if (null == value) { + return null; + } + BytesRef spare = new BytesRef(); + String stringVal = (String)value; + UnicodeUtil.UTF16toUTF8(stringVal, 0, stringVal.length(), spare); + return spare; + } + + /** + * Marshals a binary field value. + */ + protected static Object marshalBase64SortValue(Object value) { + if (null == value) { + return null; + } + final BytesRef val = (BytesRef)value; + return Base64.byteArrayToBase64(val.bytes, val.offset, val.length); + } + + /** + * Unmarshals a binary field value. + */ + protected static Object unmarshalBase64SortValue(Object value) { + if (null == value) { + return null; + } + final String val = (String)value; + final byte[] bytes = Base64.base64ToByteArray(val); + return new BytesRef(bytes); + } } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/StrField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/StrField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/StrField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/StrField.java Sat Mar 29 02:39:14 2014 @@ -29,8 +29,6 @@ import org.apache.lucene.document.Sorted import org.apache.lucene.queries.function.ValueSource; import org.apache.lucene.search.SortField; import org.apache.lucene.util.BytesRef; -import org.apache.lucene.util.CharsRef; -import org.apache.lucene.util.UnicodeUtil; import org.apache.solr.response.TextResponseWriter; import org.apache.solr.search.QParser; @@ -86,23 +84,12 @@ public class StrField extends PrimitiveF @Override public Object marshalSortValue(Object value) { - if (null == value) { - return null; - } - CharsRef spare = new CharsRef(); - UnicodeUtil.UTF8toUTF16((BytesRef)value, spare); - return spare.toString(); + return marshalStringSortValue(value); } @Override public Object unmarshalSortValue(Object value) { - if (null == value) { - return null; - } - BytesRef spare = new BytesRef(); - String stringVal = (String)value; - UnicodeUtil.UTF16toUTF8(stringVal, 0, stringVal.length(), spare); - return spare; + return unmarshalStringSortValue(value); } } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TextField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TextField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TextField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TextField.java Sat Mar 29 02:39:14 2014 @@ -23,10 +23,8 @@ import org.apache.lucene.index.Indexable import org.apache.lucene.analysis.TokenStream; import org.apache.lucene.analysis.Analyzer; import org.apache.lucene.util.BytesRef; -import org.apache.lucene.util.CharsRef; import org.apache.lucene.util.IOUtils; import org.apache.lucene.util.QueryBuilder; -import org.apache.lucene.util.UnicodeUtil; import org.apache.solr.common.SolrException; import org.apache.solr.response.TextResponseWriter; import org.apache.solr.search.QParser; @@ -175,22 +173,11 @@ public class TextField extends FieldType @Override public Object marshalSortValue(Object value) { - if (null == value) { - return null; - } - CharsRef spare = new CharsRef(); - UnicodeUtil.UTF8toUTF16((BytesRef)value, spare); - return spare.toString(); + return marshalStringSortValue(value); } @Override public Object unmarshalSortValue(Object value) { - if (null == value) { - return null; - } - BytesRef spare = new BytesRef(); - String stringVal = (String)value; - UnicodeUtil.UTF16toUTF8(stringVal, 0, stringVal.length(), spare); - return spare; + return unmarshalStringSortValue(value); } } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TrieDateField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TrieDateField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TrieDateField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/java/org/apache/solr/schema/TrieDateField.java Sat Mar 29 02:39:14 2014 @@ -81,6 +81,16 @@ public class TrieDateField extends DateF } @Override + public Object marshalSortValue(Object value) { + return value; + } + + @Override + public Object unmarshalSortValue(Object value) { + return value; + } + + @Override public ValueSource getValueSource(SchemaField field, QParser parser) { return wrappedField.getValueSource(field, parser); } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/test-files/solr/collection1/conf/schema-sorts.xml URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/test-files/solr/collection1/conf/schema-sorts.xml?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/test-files/solr/collection1/conf/schema-sorts.xml (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/test-files/solr/collection1/conf/schema-sorts.xml Sat Mar 29 02:39:14 2014 @@ -71,6 +71,94 @@ NOTE: Tests expect every field in this s + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -82,6 +170,16 @@ NOTE: Tests expect every field in this s + + + + + + + + + + @@ -94,24 +192,76 @@ NOTE: Tests expect every field in this s + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -157,6 +307,94 @@ NOTE: Tests expect every field in this s + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/CursorPagingTest.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/CursorPagingTest.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/CursorPagingTest.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/CursorPagingTest.java Sat Mar 29 02:39:14 2014 @@ -32,18 +32,21 @@ import static org.apache.solr.common.par import org.apache.solr.common.SolrException; import org.apache.solr.common.SolrException.ErrorCode; import org.apache.solr.request.SolrQueryRequest; +import org.apache.solr.schema.DateField; import org.apache.solr.search.CursorMark; //jdoc import org.noggit.ObjectBuilder; +import java.nio.ByteBuffer; import java.util.Arrays; import java.util.ArrayList; +import java.util.Date; import java.util.HashMap; import java.util.List; import java.util.Collection; import java.util.Collections; import java.util.Map; -import java.nio.ByteBuffer; +import java.util.UUID; import org.junit.BeforeClass; import org.junit.After; @@ -58,6 +61,9 @@ public class CursorPagingTest extends So public final static String TEST_SOLRCONFIG_NAME = "solrconfig-deeppaging.xml"; /** schema.xml file name, shared with other cursor related tests */ public final static String TEST_SCHEMAXML_NAME = "schema-sorts.xml"; + /** values from enumConfig.xml */ + public static final String[] SEVERITY_ENUM_VALUES = + { "Not Available", "Low", "Medium", "High", "Critical" }; @BeforeClass public static void beforeTests() throws Exception { @@ -678,7 +684,7 @@ public class CursorPagingTest extends So String cursorMark = CURSOR_MARK_START; int docsOnThisPage = Integer.MAX_VALUE; while (0 < docsOnThisPage) { - String json = assertJQ(req(params, + String json = assertJQ(req(params, CURSOR_MARK_PARAM, cursorMark)); Map rsp = (Map) ObjectBuilder.fromJSON(json); assertTrue("response doesn't contain " + CURSOR_MARK_NEXT + ": " + json, @@ -895,7 +901,6 @@ public class CursorPagingTest extends So if (useField()) { doc.addField("str", skewed(randomUsableUnicodeString(), _TestUtil.randomSimpleString(random(),1,1))); - } if (useField()) { int numBytes = (Integer) skewed(_TestUtil.nextInt(random(), 20, 50), 2); @@ -903,6 +908,23 @@ public class CursorPagingTest extends So random().nextBytes(randBytes); doc.addField("bin", ByteBuffer.wrap(randBytes)); } + if (useField()) { + doc.addField("date", skewed(randomDate(), + dateWithRandomSecondOn2010_10_31_at_10_31())); + } + if (useField()) { + doc.addField("uuid", UUID.randomUUID().toString()); + } + if (useField()) { + doc.addField("currency", skewed("" + (random().nextInt() / 100.) + "," + randomCurrency(), + "" + _TestUtil.nextInt(random(), 250, 320) + ",USD")); + } + if (useField()) { + doc.addField("bool", random().nextBoolean() ? "t" : "f"); + } + if (useField()) { + doc.addField("enum", randomEnumValue()); + } return doc; } @@ -941,6 +963,25 @@ public class CursorPagingTest extends So return result; } + private static String randomDate() { + return DateField.formatExternal(new Date(random().nextLong())); + } + + private static String dateWithRandomSecondOn2010_10_31_at_10_31() { + return String.format("2010-10-31T10:31:%02d.000Z", + _TestUtil.nextInt(random(), 0, 59)); + } + + private static final String[] currencies = { "USD", "EUR", "NOK" }; + + public static String randomCurrency() { + return currencies[random().nextInt(currencies.length)]; + } + + private static String randomEnumValue() { + return SEVERITY_ENUM_VALUES[random().nextInt(SEVERITY_ENUM_VALUES.length)]; + } + /** * Given a list of fieldNames, builds up a random sort string which is guaranteed to * have at least 3 clauses, ending with the "id" field for tie breaking @@ -958,15 +999,16 @@ public class CursorPagingTest extends So String field = shuffledNames.get(i); // wrap in a function sometimes - if ( (!"score".equals(field)) + if ( (!"score".equals(field) && !field.contains("bcd")) && (0 == _TestUtil.nextInt(random(), 0, 7)) ) { // specific function doesn't matter, just proving that we can handle the concept. // but we do have to be careful with non numeric fields - if (field.startsWith("str") || field.startsWith("bin")) { - field = "if(exists(" + field + "),47,83)"; - } else { + if (field.contains("float") || field.contains("double") + || field.contains("int") || field.contains("long")) { field = "abs(" + field + ")"; + } else { + field = "if(exists(" + field + "),47,83)"; } } result.append(field).append(random().nextBoolean() ? " asc, " : " desc, "); Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/schema/SortableBinaryField.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/schema/SortableBinaryField.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/schema/SortableBinaryField.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/schema/SortableBinaryField.java Sat Mar 29 02:39:14 2014 @@ -25,7 +25,6 @@ import org.apache.lucene.search.FieldCom import org.apache.lucene.search.FieldComparatorSource; import org.apache.lucene.search.SortField; import org.apache.lucene.util.BytesRef; -import org.apache.solr.common.util.Base64; import java.io.IOException; import java.nio.ByteBuffer; @@ -82,20 +81,11 @@ public class SortableBinaryField extends @Override public Object marshalSortValue(Object value) { - if (null == value) { - return null; - } - final BytesRef val = (BytesRef)value; - return Base64.byteArrayToBase64(val.bytes, val.offset, val.length); + return marshalBase64SortValue(value); } @Override public Object unmarshalSortValue(Object value) { - if (null == value) { - return null; - } - final String val = (String)value; - final byte[] bytes = Base64.base64ToByteArray(val); - return new BytesRef(bytes); + return unmarshalBase64SortValue(value); } } Modified: lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/search/CursorMarkTest.java URL: http://svn.apache.org/viewvc/lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/search/CursorMarkTest.java?rev=1582937&r1=1582936&r2=1582937&view=diff ============================================================================== --- lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/search/CursorMarkTest.java (original) +++ lucene/dev/branches/lucene_solr_4_7/solr/core/src/test/org/apache/solr/search/CursorMarkTest.java Sat Mar 29 02:39:14 2014 @@ -17,10 +17,15 @@ package org.apache.solr.search; +import org.apache.lucene.analysis.Analyzer; +import org.apache.lucene.analysis.TokenStream; +import org.apache.lucene.analysis.tokenattributes.TermToBytesRefAttribute; +import org.apache.lucene.util.IOUtils; import org.apache.lucene.util._TestUtil; import org.apache.lucene.util.BytesRef; import org.apache.solr.common.SolrException; import org.apache.solr.common.SolrException.ErrorCode; +import org.apache.solr.schema.DateField; import org.apache.solr.schema.IndexSchema; import org.apache.solr.schema.SchemaField; import org.apache.solr.request.SolrQueryRequest; @@ -28,11 +33,14 @@ import org.apache.solr.SolrTestCaseJ4; import org.apache.solr.CursorPagingTest; import static org.apache.solr.common.params.CursorMarkParams.CURSOR_MARK_START; +import java.io.IOException; import java.util.Arrays; import java.util.ArrayList; +import java.util.Date; import java.util.List; import java.util.Collection; import java.util.Collections; +import java.util.UUID; import org.junit.BeforeClass; @@ -51,7 +59,7 @@ public class CursorMarkTest extends Solr initCore(CursorPagingTest.TEST_SOLRCONFIG_NAME, CursorPagingTest.TEST_SCHEMAXML_NAME); } - public void testNextCursorMark() { + public void testNextCursorMark() throws IOException { final Collection allFieldNames = getAllFieldNames(); final SolrQueryRequest req = req(); final IndexSchema schema = req.getSchema(); @@ -113,7 +121,7 @@ public class CursorMarkTest extends Solr } - public void testGarbageParsing() { + public void testGarbageParsing() throws IOException { final SolrQueryRequest req = req(); final IndexSchema schema = req.getSchema(); final SortSpec ss = QueryParsing.parseSortSpec("str asc, float desc, id asc", req); @@ -160,7 +168,7 @@ public class CursorMarkTest extends Solr } } - public void testRoundTripParsing() { + public void testRoundTripParsing() throws IOException { // for any valid SortSpec, and any legal values, we should be able to round // trip serialize the totem and get the same values back. @@ -196,7 +204,7 @@ public class CursorMarkTest extends Solr } } - private static Object[] buildRandomSortObjects(SortSpec ss) { + private static Object[] buildRandomSortObjects(SortSpec ss) throws IOException { List fields = ss.getSchemaFields(); assertNotNull(fields); Object[] results = new Object[fields.size()]; @@ -225,14 +233,64 @@ public class CursorMarkTest extends Solr byte[] randBytes = new byte[_TestUtil.nextInt(random(), 1, 50)]; random().nextBytes(randBytes); val = new BytesRef(randBytes); - } else if (fieldName.startsWith("int")) { - val = (Integer) random().nextInt(); - } else if (fieldName.startsWith("long")) { - val = (Long) random().nextLong(); - } else if (fieldName.startsWith("float")) { - val = (Float) random().nextFloat() * random().nextInt(); break; - } else if (fieldName.startsWith("double")) { - val = (Double) random().nextDouble() * random().nextInt(); break; + } else if (fieldName.startsWith("bcd")) { + if (fieldName.startsWith("bcd_long")) { // BCDLongField + val = Long.toString(random().nextLong()); + val = sf.getType().toInternal((String)val); + val = sf.getType().unmarshalSortValue(val); + } else { // BCDIntField & BCDStrField + val = Integer.toString(random().nextInt()); + val = sf.getType().toInternal((String)val); + val = sf.getType().unmarshalSortValue(val); + } + } else if (fieldName.contains("int")) { + val = random().nextInt(); // TrieIntField + if (fieldName.startsWith("legacy")) { // IntField + val = Integer.toString((Integer)val); + if (fieldName.startsWith("legacy_sortable")) { // SortableIntField + val = sf.getType().unmarshalSortValue(val); + } + } + } else if (fieldName.contains("long")) { + val = random().nextLong(); // TrieLongField + if (fieldName.startsWith("legacy")) { // LongField + val = Long.toString((Long)val); + if (fieldName.startsWith("legacy_sortable")) { // SortableLongField + val = sf.getType().unmarshalSortValue(val); + } + } + } else if (fieldName.contains("float")) { + val = random().nextFloat() * random().nextInt(); // TrieFloatField + if (fieldName.startsWith("legacy")) { // FloatField + val = Float.toString((Float)val); + if (fieldName.startsWith("legacy_sortable")) { // SortableFloatField + val = sf.getType().unmarshalSortValue(val); + } + } + } else if (fieldName.contains("double")) { + val = random().nextDouble() * random().nextInt(); // TrieDoubleField + if (fieldName.startsWith("legacy")) { // DoubleField + val = Double.toString((Double)val); + if (fieldName.startsWith("legacy_sortable")) { // SortableDoubleField + val = sf.getType().unmarshalSortValue(val); + } + } + } else if (fieldName.contains("date")) { + val = random().nextLong(); // TrieDateField + if (fieldName.startsWith("legacy_date")) { // DateField + val = ((DateField)sf.getType()).toInternal(new Date((Long)val)); + val = sf.getType().unmarshalSortValue(val); + } + } else if (fieldName.startsWith("currency")) { + val = random().nextDouble(); + } else if (fieldName.startsWith("uuid")) { + val = sf.getType().unmarshalSortValue(UUID.randomUUID().toString()); + } else if (fieldName.startsWith("bool")) { + val = sf.getType().unmarshalSortValue(random().nextBoolean() ? "t" : "f"); + } else if (fieldName.startsWith("enum")) { + val = random().nextInt(CursorPagingTest.SEVERITY_ENUM_VALUES.length); + } else if (fieldName.contains("collation")) { + val = getRandomCollation(sf); } else { fail("fell through the rabbit hole, new field in schema? = " + fieldName); } @@ -243,6 +301,28 @@ public class CursorMarkTest extends Solr } return results; } + + private static Object getRandomCollation(SchemaField sf) throws IOException { + Object val = null; + Analyzer analyzer = sf.getType().getAnalyzer(); + String term = _TestUtil.randomRealisticUnicodeString(random()); + TokenStream ts = analyzer.tokenStream("fake", term); + IOException priorException = null; + try { + TermToBytesRefAttribute termAtt = ts.addAttribute(TermToBytesRefAttribute.class); + val = termAtt.getBytesRef(); + ts.reset(); + assertTrue(ts.incrementToken()); + termAtt.fillBytesRef(); + assertFalse(ts.incrementToken()); + ts.end(); + } catch (IOException e) { + priorException = e; + } finally { + IOUtils.closeWhileHandlingException(priorException, ts); + } + return val; + } /** * a list of the fields in the schema - excluding _version_