lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dingjin (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-3834) The tokenstream create by SmartChineseAnalyzer can't reset
Date Thu, 01 Mar 2012 03:56:03 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

dingjin updated LUCENE-3834:
----------------------------

       Priority: Minor  (was: Major)
    Description: 
That is because the field input in class SentenceTokenizer isn't reset after we call the method
reset().

They are two input field,one is from Tokenizer and another is from  TokenFilter,if we need
to reset a tokenstream created by SmartChineseAnalyzer, both  them need reset.This bug is
because of the author forget reset input field in class SentenceTokenizer .

class path : org.apache.lucene.analysis.cn.smart.SentenceTokenizer

oringal code

public final class SentenceTokenizer extends Tokenizer {
  ....
  @Override
  public void reset() throws IOException {
    super.reset();
    tokenStart = tokenEnd = 0;
  }

 ...
}

this method should changes as follow

 
  public void reset() throws IOException {
    super.reset();
    /*should reset input*/
    if (input.markSupported())
        input.reset();
    tokenStart = tokenEnd = 0;
  }




  was:
That is because the field input in class SentenceTokenizer isn't reset after we call the method
reset().

They are two input field,one is from Tokenizer and another is from  TokenFilter,if we need
to reset a tokenstream created by SmartChineseAnalyzer, both  them need reset.This bug is
because of the author forget reset input field in class SentenceTokenizer .

class path : org.apache.lucene.analysis.cn.smart.SentenceTokenizer

oringal code

public final class SentenceTokenizer extends Tokenizer {
  ....
  @Override
  public void reset() throws IOException {
    super.reset();
    tokenStart = tokenEnd = 0;
  }

 ...
}

this method should changes as follow

 
  public void reset() throws IOException {
    super.reset();
    /*should reset input*/
    input.reset();
    tokenStart = tokenEnd = 0;
  }





    
> The tokenstream create by SmartChineseAnalyzer can't reset
> ----------------------------------------------------------
>
>                 Key: LUCENE-3834
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3834
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: modules/analysis
>    Affects Versions: 3.5
>            Reporter: dingjin
>            Priority: Minor
>
> That is because the field input in class SentenceTokenizer isn't reset after we call
the method reset().
> They are two input field,one is from Tokenizer and another is from  TokenFilter,if we
need to reset a tokenstream created by SmartChineseAnalyzer, both  them need reset.This bug
is because of the author forget reset input field in class SentenceTokenizer .
> class path : org.apache.lucene.analysis.cn.smart.SentenceTokenizer
> oringal code
> public final class SentenceTokenizer extends Tokenizer {
>   ....
>   @Override
>   public void reset() throws IOException {
>     super.reset();
>     tokenStart = tokenEnd = 0;
>   }
>  ...
> }
> this method should changes as follow
>  
>   public void reset() throws IOException {
>     super.reset();
>     /*should reset input*/
>     if (input.markSupported())
>         input.reset();
>     tokenStart = tokenEnd = 0;
>   }

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message