lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-1916) investigate DIH use of default locale
Date Wed, 07 Nov 2012 22:42:12 GMT

    [ https://issues.apache.org/jira/browse/SOLR-1916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492775#comment-13492775
] 

Robert Muir commented on SOLR-1916:
-----------------------------------

James, thanks for looking at this!!!

It may not be a locale issue, instead a time zone issue (or both). But this test definitely
failed intermittently in the past.

For example, it failed during a Daylight savings time window (but only for developers in Europe!)
and Chris Male addressed
some of the issues in SOLR-1821.

Fortunately, Uwe Schindler has made it dead easy to identify most of these issues: we not
longer have to solely rely upon unit tests alone.

http://blog.thetaphi.de/2012/07/default-locales-default-charsets-and.html

DIH currently has 40 violations!

Try this:
{noformat}
Index: build.xml
===================================================================
--- build.xml	(revision 1406757)
+++ build.xml	(working copy)
@@ -250,8 +250,6 @@
       </apiFileSet>
       <fileset dir="${basedir}/build">
         <include name="**/*.class" />
-        <!-- exclude DIH for now as it is broken with Locales and Encodings: SOLR-1916
-->
-        <exclude name="contrib/solr-dataimporthandler*/**" />
       </fileset>
     </forbidden-apis>
   </target>
{noformat}

Then run
{noformat}
rmuir@beast:~/workspace/lucene-trunk/solr$ ant check-forbidden-apis
...
-check-forbidden-java-apis:
[forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/commons-io.txt
[forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/executors.txt
[forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/jdk-deprecated.txt
[forbidden-apis] Reading API signatures: /home/rmuir/workspace/lucene-trunk/lucene/tools/forbiddenApis/jdk.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning for API signatures and dependencies...
[forbidden-apis] Forbidden method invocation: java.text.DecimalFormatSymbols#<init>()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestNumberFormatTransformer (TestNumberFormatTransformer.java:36)
[forbidden-apis] Forbidden method invocation: java.text.DecimalFormatSymbols#<init>()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestNumberFormatTransformer (TestNumberFormatTransformer.java:37)
[forbidden-apis] Forbidden method invocation: java.text.MessageFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.DebugLogger (DebugLogger.java:52)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestDateFormatTransformer (TestDateFormatTransformer.java:43)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestDateFormatTransformer (TestDateFormatTransformer.java:66)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.MailEntityProcessor (MailEntityProcessor.java:88)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:250)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:251)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:252)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestDocBuilder2 (TestDocBuilder2.java:257)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.DataImporter$3 (DataImporter.java:490)
[forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[])
[forbidden-apis]   in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:711)
[forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[])
[forbidden-apis]   in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:717)
[forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[])
[forbidden-apis]   in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:725)
[forbidden-apis] Forbidden method invocation: java.lang.String#format(java.lang.String,java.lang.Object[])
[forbidden-apis]   in org.apache.solr.handler.dataimport.DocBuilder (DocBuilder.java:727)
[forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream)
[forbidden-apis]   in org.apache.solr.handler.dataimport.FieldReaderDataSource (FieldReaderDataSource.java:109)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor
(TestFileListWithLineEntityProcessor.java:36)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor
(TestFileListWithLineEntityProcessor.java:37)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListWithLineEntityProcessor
(TestFileListWithLineEntityProcessor.java:38)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:92)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:101)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:107)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:138)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestVariableResolver (TestVariableResolver.java:147)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.EvaluatorBag$4 (EvaluatorBag.java:164)
[forbidden-apis] Forbidden method invocation: java.util.GregorianCalendar#<init>()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:122)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:125)
[forbidden-apis] Forbidden method invocation: java.util.GregorianCalendar#<init>()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:128)
[forbidden-apis] Forbidden method invocation: java.text.SimpleDateFormat#<init>(java.lang.String)
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestEvaluatorBag (TestEvaluatorBag.java:135)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:43)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:44)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:45)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:140)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:141)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:142)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:164)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:179)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:180)
[forbidden-apis] Forbidden method invocation: java.lang.String#getBytes()
[forbidden-apis]   in org.apache.solr.handler.dataimport.TestFileListEntityProcessor (TestFileListEntityProcessor.java:181)
[forbidden-apis] Forbidden method invocation: java.io.InputStreamReader#<init>(java.io.InputStream)
[forbidden-apis]   in org.apache.solr.handler.dataimport.FileDataSource (FileDataSource.java:128)
[forbidden-apis] Scanned 2080 (and 648 related) class file(s) for forbidden API invocations
(in 1.08s), 40 error(s).

BUILD FAILED
/home/rmuir/workspace/lucene-trunk/solr/build.xml:243: Check for forbidden API calls failed,
see log.
{noformat}
                
> investigate DIH use of default locale
> -------------------------------------
>
>                 Key: SOLR-1916
>                 URL: https://issues.apache.org/jira/browse/SOLR-1916
>             Project: Solr
>          Issue Type: Task
>          Components: contrib - DataImportHandler
>    Affects Versions: 3.1, 4.0-ALPHA
>            Reporter: Robert Muir
>            Assignee: Robert Muir
>             Fix For: 4.1
>
>         Attachments: SOLR-1916.patch
>
>
> This is a spinoff from LUCENE-2466.
> In this issue I changed my locale to various locales and found some problems in Lucene/Solr
triggered by use of the default Locale.
> I noticed some use of the default-locale for Date operations in DIH (TimeZone.getDefault/Locale.getDefault)
and, while no tests fail, I think it might be better to support a locale parameter for this.
> The wiki documents that numeric parsing can support localized numerics formats: http://wiki.apache.org/solr/DataImportHandler#NumberFormatTransformer
> In both cases, I don't think we should ever use the default Locale. If no Locale is provided,
I find that new Locale("") <-- Unicode Root Locale, is a better default for a server situation
in a lot of cases, as it won't change depending on the computer, or perhaps we just make Locale
params mandatory for this.
> Finally, in both cases, if localized numbers/dates are explicitly supported, I think
we should come up with a test strategy to ensure everything is working. One idea is to do
something similar to or make use of Lucene's LocalizedTestCase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message