lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Muir (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-2839) add alternative language detection impl
Date Sun, 16 Oct 2011 03:16:11 GMT

    [ https://issues.apache.org/jira/browse/SOLR-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128335#comment-13128335
] 

Robert Muir commented on SOLR-2839:
-----------------------------------

ok, i'd like to add this basic implementation first.

later, we should add support for some advanced parameters and refactoring:
* whitelisting should not happen in the base class as a post-filter (though this is fine as
a default implementation), but subclasses should override i think. For this detector, it could
improve performance.
* for this detector whitelist should support priors too (e.g. en=0.5, fr=0.1).
* we should add support for configuring smoothing parameter and maxTextLength (and, the base
class's concat should respect that too).
* both this implementation and the tika implementation are copying objects across lists of
language information, i think this is not very efficient to do per-document. So I think we
should change the API from List<DetectedLanguage> detectLanguage() to Iterable<DetectedLanguage>
detectLanguage. It seems in general it just wants the first one anyway.
                
> add alternative language detection impl
> ---------------------------------------
>
>                 Key: SOLR-2839
>                 URL: https://issues.apache.org/jira/browse/SOLR-2839
>             Project: Solr
>          Issue Type: Improvement
>            Reporter: Robert Muir
>         Attachments: SOLR-2839.patch
>
>
> based on http://code.google.com/p/language-detection (apache license), supports 53 languages.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message