lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Krajňanský <>
Subject Custom TokenStream + custom Attributes
Date Tue, 31 May 2016 13:58:45 GMT
Dear Lucene users,

I have implemented a custom tokenizer (derived from TokenStream).

I need to pass additional attributes to those standard in Lucene
(PositionIncrementAttribute, OffsetAttribute), that would represent the
word position in the tokenized sentence in the number of words and not
characters, as one usually passes through OffsetAttribute. (I need both.)

Is there a way of achieving this?

I tried to implement own Attribute class (derive a new interface and
implementing class). The code compiles ok but I am getting exception at
runtime about the class casting.

Thank you a lot in advance,


FYI the code looks like this:

package com.newstin.nlp.analysis;

import java.util.Iterator;
import java.util.List;

import org.apache.lucene.analysis.TokenStream;
import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;

 * @author michal
public class TermsListTokenizer extends TokenStream
    private final CharTermAttribute termAtt =
    private final OffsetAttribute offsetAtt =
    private final PositionIncrementAttribute posIncrAtt =

    private final Iterator<Term> termIterator;
    private int lastTermPos;

    public TermsListTokenizer(List<Term> terms)
        termIterator = terms.iterator();
        lastTermPos = -1;

    public boolean incrementToken() throws IOException

        // TODO: check: compute the positions right for term variants !!!
        if (termIterator.hasNext()) {
            Term term =;

            offsetAtt.setOffset(term.getStart(), term.getEnd()); // need to
also save position in the number of words
            posIncrAtt.setPositionIncrement(term.getWordIndex() -
            lastTermPos = term.getWordIndex();
            return true;

        return false;

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message