lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-1377) Add HTMLStripReader and WordDelimiterFilter from SOLR
Date Wed, 03 Jun 2009 03:26:07 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715786#action_12715786
] 

Robert Muir commented on LUCENE-1377:
-------------------------------------

i don't really know enough to say that, but this one is especially important (imho)

without getting into details, the changes Yonik made to WordDelimiterFilter, in combination
with WhitespaceTokenizer treat the various unicode categories correctly, unlike any of the
other analyzers in lucene. 

No, it doesn't actually use the Word_Break properties which is really the key, but if you
toss some Hindi, Bangla, Tibetan, Arabic, Tamil, ... [list goes on very long] text at it,
in general it's gonna work pretty well. This is a big improvement over any of the other default
options!


> Add HTMLStripReader and WordDelimiterFilter from SOLR
> -----------------------------------------------------
>
>                 Key: LUCENE-1377
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1377
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Analysis
>    Affects Versions: 2.3.2
>            Reporter: Jason Rutherglen
>            Priority: Minor
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> SOLR has two classes HTMLStripReader and WordDelimiterFilter which are very useful for
a wide variety of use cases.  It would be good to place them into core Lucene.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message