Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Wed, 13 Feb 2013 13:32:13 +0000 (UTC)
From: "Clinton Gormley (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <JIRA.12631641.1360499498778.283408.1360762333150@arcas>
In-Reply-To: <JIRA.12631641.1360499498778@arcas>
References: <JIRA.12631641.1360499498778@arcas>
Subject: [jira] [Updated] (LUCENE-4766) Pattern token filter which emits a
 token for every capturing group
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/LUCENE-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Clinton Gormley updated LUCENE-4766:
------------------------------------

    Attachment: LUCENE-4766.patch

The charOffsetEnd now uses the correct offsetAttr.endOffset() and, added a test using checkRandomData()
                
> Pattern token filter which emits a token for every capturing group
> ------------------------------------------------------------------
>
>                 Key: LUCENE-4766
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4766
>             Project: Lucene - Core
>          Issue Type: New Feature
>          Components: modules/analysis
>    Affects Versions: 4.1
>            Reporter: Clinton Gormley
>            Assignee: Simon Willnauer
>            Priority: Minor
>              Labels: analysis, feature, lucene
>             Fix For: 4.2
>
>         Attachments: LUCENE-4766.patch, LUCENE-4766.patch, LUCENE-4766.patch, LUCENE-4766.patch
>
>
> The PatternTokenizer either functions by splitting on matches, or allows you to specify a single capture group.  This is insufficient for my needs. Quite often I want to capture multiple overlapping tokens in the same position.
> I've written a pattern token filter which accepts multiple patterns and emits tokens for every capturing group that is matched in any pattern.
> Patterns are not anchored to the beginning and end of the string, so each pattern can produce multiple matches.
> For instance a pattern like :
> {code}
>     "(([a-z]+)(\d*))"
> {code}
> when matched against: 
> {code}
>     "abc123def456"
> {code}
> would produce the tokens:
> {code}
>     abc123, abc, 123, def456, def, 456
> {code}
> Multiple patterns can be applied, eg these patterns could be used for camelCase analysis:
> {code}
>     "([A-Z]{2,})",
>     "(?<![A-Z])([A-Z][a-z]+)",
>     "(?:^|\\b|(?<=[0-9_])|(?<=[A-Z]{2}))([a-z]+)",
>     "([0-9]+)"
> {code}
> When matched against the string "letsPartyLIKEits1999_dude", they would produce the tokens:
> {code}
>     lets, Party, LIKE, its, 1999, dude
> {code}
> If no token is emitted, the original token is preserved. 
> If the preserveOriginal flag is true, it will output the full original token (ie "letsPartyLIKEits1999_dude") in addition to any matching tokens (but in this case, if a matching token is identical to the original, it will only emit one copy of the full token).
> Multiple patterns are required to allow overlapping captures, but also means that patterns are less dense and easier to understand.
> This is my first Java code, so apologies if I'm doing something stupid.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org