lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Webster Homer <>
Subject Filter Factory question
Date Tue, 26 Sep 2017 21:14:41 GMT
I am trying to create a filter that normalizes an input token, but also
splits it inot multiple pieces. Sort of like what the WordDelimiterFilter

It's meant to take a molecular formula like C2H6O and normalize it to C2H6O1

That part works. However I was also going to have it put out the individual
atom counts as tokens.

When I enable this feature in the factory, I don't get any output at all.

I looked over a couple of filters that do what I want and it's not entirely
clear what they're doing. So I have some questions:
Looking at ShingleFilter and WordDelimitierFilter
They both set several attributes:
CharTermAttribute : Seems to be the actual terms being set. Seemed straight
forward, works fine when I only have one term to add.

PositionIncrementAttribute: What does this do? It appears that
WordDelimiterFilter sets this to 0 most of the time. This has decent

OffsetAttribute: I think that this tracks offsets for each term being
processed. Not really sure though. The documentation mentions tokens. So if
I have multiple variations for for a token is this for each variation?

TypeAttribute: default is "word". Don't know what this is for.

PositionLengthAttribute: WordDelimiterFilter doesn' use this but Shingle
does. It defaults to 1. What's it good for when should I use it?

Here is my incrementToken method.

    public boolean incrementToken() throws IOException {
    while(true) {
    if (!hasSavedState) {
    if (! input.incrementToken()) {
    return false;
    if (! generateFragments) { // This part works fine!
        String normalizedFormula = molFormula.normalize(new
        char[]newBuffer = normalizedFormula.toCharArray();
        termAttribute.copyBuffer(newBuffer, 0, newBuffer.length);
        return true;
    formulas = molFormula.normalizeToList(new
    iterator = formulas.listIterator();
    savedPositionIncrement += posIncAttribute.getPositionIncrement();
    hasSavedState = true;
    first = true;
    if (!iterator.hasNext()) {
    savedPositionIncrement = 0;
    hasSavedState = false;
    String formula =;
        int startOffset = savedStartOffset;

        if (first) {
        int endOffset = savedStartOffset + formula.length();
        System.out.printf("Writing formula %s %d to %d%n", formula,
startOffset, endOffset);;
            offsetAttribute.setOffset(startOffset, endOffset);
            savedStartOffset = endOffset + 1;
            if (first) {
            } else {
            first = false;
            return true;


This message and any attachment are confidential and may be privileged or 
otherwise protected from disclosure. If you are not the intended recipient, 
you must not copy this message or attachment or disclose the contents to 
any other person. If you have received this transmission in error, please 
notify the sender immediately and delete the message and any attachment 
from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not accept liability for any omissions or errors in this 
message which may arise as a result of E-Mail-transmission or for damages 
resulting from any unauthorized changes of the content of this message and 
any attachment thereto. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee that this message is free of viruses and does 
not accept liability for any damages caused by any virus transmitted 

Click to access the German, French, 
Spanish and Portuguese versions of this disclaimer.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message