lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tomoko Uchida (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LUCENE-8817) Combine Nori and Kuromoji DictionaryBuilder
Date Sun, 09 Jun 2019 02:46:00 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16859344#comment-16859344
] 

Tomoko Uchida edited comment on LUCENE-8817 at 6/9/19 2:45 AM:
---------------------------------------------------------------

For me, it looks like a good starting point to create a directory {{analysis/mecab}} and place
{{mecab-tools}} module (the option 4) under that.

We are already considering further integration of kuromoij and nori (at LUCENE-8816 and LUCENE-8812),
and I suppose it would happen sooner or later. So how about looking at some grand design from
here. For example:
{code:java}
analysis
└── mecab
         ├── common (module: analyzers-mecab-common)
         │       ├── build.xml
         │       └── src
         ├── kuromoji (module: analyzers-mecab-kuromoji)
         │       ├── build.xml
         │       └── src
         ├── nori (module: analyzers-mecab-nori)
         │       ├── build.xml
         │       └── src
         └── tools  (module: analyzers-mecab-tools)
                 ├── build.xml
                 └── src
{code}
On this issue, only "mecab-tools" module will be added (and the dependency on that should
be added to current kuromoji and nori).
 That's just an idea and I am not an expert about the shadow maven poms. [~rcmuir], [~jim.ferenczi]
and [~cm] may have different thoughts.

About the option 2, I don't think that's a good idea to change other modules' current structure
(analysis-common or icu), opinions?
{quote}I will go ahead if direction is set, but landing will be delayed a little.
 The reason is that the build system is going to change. (SOLR-13452)
 But if it does not matter, I will proceed.
{quote}
We need to take care the ongoing change in build infrastructure, but I think it would not
be a very big concern that stops the work here (and LUCENE-8816) :) After pushing the commits
to the master, you would be able to backport the changes to the gradle branch (I think Mark
Miller and others will give you help or advice for the work).


was (Author: tomoko uchida):
For me, it looks like a good starting point to create a directory {{analysis/mecab}} and place
{{mecab-tools}} module (the option 4) under that.

We are already considering further integration of kuromoij and nori (at LUCENE-8816 and LUCENE-8812),
and I suppose it would happen sooner or later. So how about looking at some ground design
from here. For example:
{code:java}
analysis
└── mecab
         ├── common (module: analyzers-mecab-common)
         │       ├── build.xml
         │       └── src
         ├── kuromoji (module: analyzers-mecab-kuromoji)
         │       ├── build.xml
         │       └── src
         ├── nori (module: analyzers-mecab-nori)
         │       ├── build.xml
         │       └── src
         └── tools  (module: analyzers-mecab-tools)
                 ├── build.xml
                 └── src
{code}
On this issue, only "mecab-tools" module will be added (and the dependency on that should
be added to current kuromoji and nori).
 That's just an idea and I am not an expert about the shadow maven poms. [~rcmuir], [~jim.ferenczi]
and [~cm] may have different thoughts.

About the option 2, I don't think that's a good idea to change other modules' current structure
(analysis-common or icu), opinions?
{quote}I will go ahead if direction is set, but landing will be delayed a little.
 The reason is that the build system is going to change. (SOLR-13452)
 But if it does not matter, I will proceed.
{quote}
We need to take care the ongoing change in build infrastructure, but I think it would not
be a very big concern that stops the work here (and LUCENE-8816) :) After pushing the commits
to the master, you would be able to backport the changes to the gradle branch (I think Mark
Miller and others will give you help or advice for the work).

> Combine Nori and Kuromoji DictionaryBuilder
> -------------------------------------------
>
>                 Key: LUCENE-8817
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8817
>             Project: Lucene - Core
>          Issue Type: New Feature
>            Reporter: Namgyu Kim
>            Priority: Major
>
> This issue is related to LUCENE-8816.
> Currently Nori and Kuromoji Analyzer use the same dictionary structure. (MeCab)
>  If we make combine DictionaryBuilder, we can reduce the code size.
>  But this task may have a dependency on the language.
>  (like HEADER string in BinaryDictionary and CharacterDefinition, methods in BinaryDictionaryWriter,
...)
>  On the other hand, there are many overlapped classes.
> The purpose of this patch is to provide users of Nori and Kuromoji with the same system
dictionary generator.
> It may take some time because there is a little workload.
>  The work will be based on the latest master, and if the LUCENE-8816 is finished first, I
will pull the latest code and proceed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message