lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ziqi Zhang <ziqi.zh...@sheffield.ac.uk>
Subject Re: lucene deliberately removes \r (windows carriage char)
Date Sat, 03 Oct 2015 17:39:25 GMT
Well this is very strange then. If I knew where exactly those 
"IndexableField" are constructed in the pipeline i could possibly pin 
down the bug...

In any case,  no I did not use MappingCharFilter or a BufferedReader. 
The way I pass content to analyse is straightforward:
 >>>
SolrInputDocument solrDoc = new SolrInputDocument();
solrDoc.addField("content", "ok\r\nhere is the text\r\n");
......


The schema for the field "content" to be analysed begins with taking the 
text content in the field for tokenization:
 >>>
<analyzer type="index">
                 <tokenizer 
class="org.apache.lucene.analysis.opennlp.OpenNLPTokenizerFactory"
                             sentenceModel="en-sent.bin"
                             tokenizerModel="en-token.bin"/>
.............


Where OpenNLPTokenizerFactory creates a OpenNLPTokenizer, what is 
identical to the code provide at
https://issues.apache.org/jira/browse/LUCENE-2899
except that I adapted to Lucene 5.3

And by looking at the source code of OpenNLPTokenizer, I can see it is 
using the "input" variable (type Reader) of the superclass Tokenizer to 
get the text content to be analyzed. At runtime through debugging I see 
that "input" is instantiated as a "ReusableStringReader", and you can 
see the string value has become "ok\nhere is the text\n"

Any other thoughts please?


On 03/10/2015 17:37, Michael McCandless wrote:
> Are you using MappingCharFilter?
>
> It unfortunately has known bugs which require controversial API
> changes to fix: https://issues.apache.org/jira/browse/LUCENE-6595
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
> On Sat, Oct 3, 2015 at 6:02 PM, Uwe Schindler <uwe@thetaphi.de> wrote:
>> Hi,
>>
>> Lucene does not remove the \r\n while indexing or storing fields. The Analyzer just
splits e.g., at whitespace (depends on Analyzer). So if you original data has \r\n, then the
offsets would be according to that (it counts 2 chars).
>>
>> Could it be that you read it using a BufferedReader per line and pass as Strings?
>>
>> Uwe
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>>> -----Original Message-----
>>> From: Ziqi Zhang [mailto:ziqi.zhang@sheffield.ac.uk]
>>> Sent: Saturday, October 03, 2015 5:01 PM
>>> To: java-user@lucene.apache.org
>>> Subject: lucene deliberately removes \r (windows carriage char)
>>>
>>> Hi
>>>
>>> I am trying to pin-point a mismatch between the offsets produced by lucene
>>> indexing process when I use the offsets to substring from the original
>>> document content.
>>>
>>> I try to debug as far as I can go but I lost track of lucene when I am at line
298
>>> of DefaultIndexingChain (lucene 5.3.0):
>>>
>>> for (IndexableField field : docState.doc) {
>>>           fieldCount = processField(field, fieldGen, fieldCount);
>>>         }
>>>
>>> Basically at this point I can see that the content field (one of the
>>> IndexableField) I am interested in has already removed all "\r" from the
>>> "\r\n" newline characters (windows) from the content. But I am unable to
>>> trace how these IndexableField are generated, and how the raw content is
>>> passed to them.
>>>
>>> I can be certain that my program did pass strings with lots of "\r\n"
>>>
>>> So the question is is this (i.e., removing \r) deliberate?
>>>
>>> Thanks
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>


-- 
Ziqi Zhang
Research Associate
Department of Computer Science
University of Sheffield


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message