lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] [Commented] (LUCENE-7465) Add a PatternTokenizer that uses Lucene's RegExp implementation
Date Mon, 26 Sep 2016 15:01:20 GMT


Michael McCandless commented on LUCENE-7465:

bq.  default to current java regexp impl;

But I think that would mean this new impl would very rarely be used.

I think it's better to give it a separate factory so it has more visibility?  If it really
does work better for users over time, word will spread, new blog posts/docs written, etc.

> Add a PatternTokenizer that uses Lucene's RegExp implementation
> ---------------------------------------------------------------
>                 Key: LUCENE-7465
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: master (7.0), 6.3
>         Attachments: LUCENE-7465.patch, LUCENE-7465.patch
> I think there are some nice benefits to a version of PatternTokenizer that uses Lucene's
RegExp impl instead of the JDK's:
>   * Lucene's RegExp is compiled to a DFA up front, so if a "too hard" RegExp is attempted
the user discovers it up front instead of later on when a "lucky" document arrives
>   * It processes the incoming characters as a stream, only pulling 128 characters at
a time, vs the existing {{PatternTokenizer}} which currently reads the entire string up front
(this has caused heap problems in the past)
>   * It should be fast.
> I named it {{SimplePatternTokenizer}}, and it still needs a factory and improved tests,
but I think it's otherwise close.
> It currently does not take a {{group}} parameter because Lucene's RegExps don't yet implement
sub group capture.  I think we could add that at some point, but it's a bit tricky.
> This doesn't even have group=-1 support (like String.split) ... I think if we did that
we should maybe name it differently ({{SimplePatternSplitTokenizer}}?).

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message