lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <>
Subject Re: Installing a custom tokenizer
Date Tue, 29 Aug 2006 23:12:27 GMT
Bill Taylor wrote:
> I have copied Lucene's StandardTokenizer.jj into my directory, renamed 
> it, and did a global change of the names to my class name, LogTokenizer.
> The issue is that the generated does not compile for 
> 2 reasons:
> 1) in the constructor, this(new FastCharStream(reader)); fails because 
> there is no such constructor in the parent class.  I commented it out.
> 2) I get an error on the next() method which throws ParseException and 
> IO Exception.  The message is Exception ParseException is not 
> compatible with throws clause in  As far as I can 
> see, the exceptions are OK.
> Since all of this is generated code, my feelings are a bit hurt.  Did 
> Lucene use an older version of JavaCC?  I am using javacc-4.0
> On Aug 29, 2006, at 4:57 PM, Erick Erickson wrote:
Ok. How about some better answers:

1. this(new FastCharStream(reader)) does not refer to the parent class 
but the LogTokenizer class itself. There should be a constructor on the 
LogTokenizer that takes a CharStream:

  public StandardTokenizer(CharStream stream) {
    token_source = new StandardTokenizerTokenManager(stream);
    token = new Token();
    jj_ntk = -1;
    jj_gen = 0;
    for (int i = 0; i < 1; i++) jj_la1[i] = -1;

If you're LogTokenizer does not have this then something did not go right.

2. The ParseException that is generated when making the StandardAnalyzer 
must be killed because there is another ParseException class (maybe in 
queryparser?) that must be used instead. The lucene build file excludes 
the StandardAnalyzer ParseException so that the other one is used. You 
could prob just delete it as well but then of course you would have to 
remember to delete it every time you rebuilt the javacc file.

- Mark

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message