lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Namgyu Kim (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-8817) Combine Nori and Kuromoji DictionaryBuilder
Date Fri, 07 Jun 2019 18:41:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16858895#comment-16858895
] 

Namgyu Kim commented on LUCENE-8817:
------------------------------------

I share the current status.
 The merge is almost over and I need some discussion.

 

I thought several structures.

1. Save in tools of analysis-common module.
 It is simple, but I think MeCab is difficult to see as a feature of analysis-common.

2. Create tools folder in analysis and set mecab-tools module in there.
 analysis/tools ─ analysis-common-tools (to-be)
                      └ icu-tools (to-be)
                      └ mecab-tools
                      └ ...
 The problem with this is that the number of modules increases a lot because each tool is
created as a module.

3. Create a module called mecab
 we can create a mecab module that is the starting point for merging nori and kuromoji.
 If we proceed in this direction, we will only have tools in src.

But this approach may not be easy to create the runnable jar.
 Because it will include the library.
 (ex: MecabAnalyzer, MecabTokenizer, ...)

4. Create a module called mecab-tools
 It's easy to develop, but there are other library modules in analysis.
 So something seems strange because it's only runnable-jar.

 

Number 2 seems to be the best, but I'm not sure yet.
 I would appreciate any comments.

 

I will go ahead if direction is set, but landing will be delayed a little.
 The reason is that the build system is going to change. (SOLR-13452)
 But if it does not matter, I will proceed.

> Combine Nori and Kuromoji DictionaryBuilder
> -------------------------------------------
>
>                 Key: LUCENE-8817
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8817
>             Project: Lucene - Core
>          Issue Type: New Feature
>            Reporter: Namgyu Kim
>            Priority: Major
>
> This issue is related to LUCENE-8816.
> Currently Nori and Kuromoji Analyzer use the same dictionary structure. (MeCab)
>  If we make combine DictionaryBuilder, we can reduce the code size.
>  But this task may have a dependency on the language.
>  (like HEADER string in BinaryDictionary and CharacterDefinition, methods in BinaryDictionaryWriter,
...)
>  On the other hand, there are many overlapped classes.
> The purpose of this patch is to provide users of Nori and Kuromoji with the same system
dictionary generator.
> It may take some time because there is a little workload.
>  The work will be based on the latest master, and if the LUCENE-8816 is finished first, I
will pull the latest code and proceed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message