camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From furchess123 <>
Subject Re: correct way to provide regex in TokenizerExpression?
Date Thu, 29 Oct 2015 22:38:35 GMT
Ok, here's the workaround I have implemented to go past the above issue...

Some file:

    public static final String SYSTEM_AGNOSTIC_NEWLINE_REGEX = "\r|\r\n|\n";

Splitter route configuration in a RouteBuilder implementation:

       TokenizerExpression tokenizerExpression = new TokenizerExpression();
tokenizerExpression.setToken(MyConstants.SYSTEM_AGNOSTIC_NEWLINE_REGEX);  //
tokenize by line separators
group so many lines into one exchange
        tokenizerExpression.setRegex(true);  // indicate that it is a
regular expression, not simple string match

                streaming().          // enable streaming vs. reading all
into memory
                parallelProcessing(readerConfig.isParallelProcessing()). //
on/off concurrent processing of multiple chunks
                stopOnException().    // stop processing file if system
exception occurs (handled by onException clause)
                *bean(new TokenizerCharRemover())*. // cleans junk chars
inserted by Camel's tokenizer due to bug(?) 
                unmarshal().csv().    // unmarshal each chunk to Java (list
of String lists) using Camel's CSV component
                bean(csvHandler).     // hand each unmarshalled list of
lines/fields to bean that parses and validates line content
                bean(importProcessor).// process codes for import (depending
on operational mode and errors in exchange)
                to(AGGREGATE_ERRORS_ENDPOINT).      // delegate to nested
route to update error report

public class TokenizerCharRemover
     * Pre-compiled regex pattern to match the instances of character
sequences of the regular expression inserted by
     * Camel's splitter's tokenizer between the file lines in the body of
the exchange.  The input string that specifies
     * the pattern is treated as a sequence of literal characters thanks to
the {@link Pattern#LITERAL} flag.
    private static final Pattern REPLACE_JUNK_PATTERN =

     * Replaces every instance of the {@link
FileContext#SYSTEM_AGNOSTIC_NEWLINE_REGEX} character sequence in the
     * exchange body with a simple '\n' line separator.
    public void cleanupLineSeparators(Exchange exchange)
        String newBody =


If there is a better solution, or if I have missed some obvious simple way
to use the tokenizer that does not replace the matching line separators with
the regex character sequence itself, please let me know! I'd very much
appreciate that.

View this message in context:
Sent from the Camel - Users mailing list archive at

View raw message