Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of paul_t100@fastmail.fm
 designates 66.111.4.29 as permitted sender)
Message-ID: <4E9D3F6B.6080009@fastmail.fm>
Date: Tue, 18 Oct 2011 10:57:15 +0200
From: Paul Taylor <paul_t100@fastmail.fm>
Reply-To: paul_t100@fastmail.fm
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
 rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1
MIME-Version: 1.0
CC: "java-user@lucene.apache.org" <java-user@lucene.apache.org>
Subject: Re: How do you see if a tokenstream has tokens without consuming
 the tokens ?
References: <4E9C1BCB.7080900@fastmail.fm>
 <6C78E97C707B5B4C8CC61D44F8754586032615@SUEX10-mbx-03.ad.syr.edu>
In-Reply-To: <6C78E97C707B5B4C8CC61D44F8754586032615@SUEX10-mbx-03.ad.syr.edu>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 18/10/2011 06:19, Steven A Rowe wrote:On 18/10/2011 06:19, Steven A 
Rowe wrote:
> Hi Paul,
>
> You could add a rule to the StandardTokenizer JFlex grammar to handle 
> this case, bypassing its other rules.
Hmm, dont really understand jflex, but that is a possibility, but would 
prefer to do in Java code unless easy to use jflex
> Another option is to create a char filter that substitutes 
> PUNCT-EXCLAMATION for exclamation points, PUNCT-PERIOD for periods, etc.,

Yes that is how I first did it
> but only when the entire input consists exclusively of whitespace and 
> punctuation.

but I couldnt work out how to only do it when exclusively whitespace and 
punctuation, any ideas to sole that _
>   These symbols would then be left intact by StandardTokenizer.
>
> Steve
>
Paul

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org