lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Rowe (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (LUCENE-3913) HTMLStripCharFilter produces invalid final offset
Date Sat, 24 Mar 2012 20:04:24 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237648#comment-13237648
] 

Steven Rowe edited comment on LUCENE-3913 at 3/24/12 8:03 PM:
--------------------------------------------------------------

Patch, a superset of Mike's:

* fixes the identified problem: {{</br>}} offset was improperly calculated.  (Added
comments describing the offset calculations everywhere they're performed in the .jflex source.)
* adds a new case emitting {{<\s*(/\s*)?(br|script|style)>?}} to {{_TestUtil.randomHtmlishString()}},
because <br>, <script>, and <style> are handled specially in HTMLStripCharFilter.
* adds a new method {{_TestUtil.randomlyRecaseCodePoints()}}, used by the above-mentioned
new {{randomHtmlishString()}} case, to produce things like {{<Br>}}, {{</sCriPT>}},
etc.
* switches {{HTMLStripCharFilterTest.testRandomBrokenHTML()}} to use Mike's new {{BaseTokenStreamTestCase.checkAnalysisConsistency()}}.
* fixes the Jenkins test failure of {{HTMLStripCharFilterTest.testRandomHugeStrings()}} at
[https://builds.apache.org/job/Lucene-Solr-tests-only-3.x/12863/]:

{noformat}
ant test -Dtestcase=HTMLStripCharFilterTest -Dtestmethod=null -Dtests.seed=48bbf57c15b7aa2d:5bb640584c81078d:-7e916259eafd7e54
-Dtests.multiplier=5 -Dargs="-Dfile.encoding=ISO8859-1"
{noformat}

Committing shortly.
                
      was (Author: steve_rowe):
    Patch, a superset of Mike's:

* fixes the identified problem: {{</br>}} offset was improperly calculated.  (Added
comments describing the offset calculations everywhere they're performed in the .jflex source.)
* adds a new case emitting {{<\s*(/\s*)?(br|script|style)>?}} to {{_TestUtil.randomHtmlishString()}},
because <br>, <script>, and <style> are handled specially in HTMLStripCharFilter.
* adds a new method {{_TestUtil.randomlyRecaseCodePoints()}}, used by the above-mentioned
new {{randomHtmlishString()}} case, to produce things like {{<Br>}}, {{</sCriPT>}},
etc.
* switches {{HTMLStripCharFilterTest.testRandomBrokenHTML()}} to use Mike's new {{BaseTokenStreamTestCase.checkAnalysisConsistency()}}.
* fixes the Jenkins test failure of {{HTMLStripCharFilterTest.testRandomHugeStrings()}} at
[https://builds.apache.org/job/Lucene-Solr-tests-only-3.x/12863/]: {{ant test -Dtestcase=HTMLStripCharFilterTest
-Dtestmethod=null -Dtests.seed=48bbf57c15b7aa2d:5bb640584c81078d:-7e916259eafd7e54 -Dtests.multiplier=5
-Dargs="-Dfile.encoding=ISO8859-1"
}}

Committing shortly.
                  
> HTMLStripCharFilter produces invalid final offset
> -------------------------------------------------
>
>                 Key: LUCENE-3913
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3913
>             Project: Lucene - Java
>          Issue Type: Bug
>            Reporter: Michael McCandless
>            Assignee: Steven Rowe
>             Fix For: 3.6, 4.0
>
>         Attachments: LUCENE-3913.patch, LUCENE-3913.patch
>
>
> Nightly build found this... I boiled it down to a small test case that doesn't require
the big line file docs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message