Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of
 SRS0=y3ZcDS=SD=basetechnology.com=jack@yourhostingaccount.com designates
 65.254.254.70 as permitted sender)
Message-ID: <566EDD43E7BE446AB3233A290EDF5348@JackKrupansky>
From: "Jack Krupansky" <jack@basetechnology.com>
To: <solr-user@lucene.apache.org>
References: 
 <CAH=GueQL2cXCiU3P73kmXaqmZ=kHUo=9Jjxkw1+cFXbwcwxbPw@mail.gmail.com>
 <52157A3C.9010909@elyograg.org>
In-Reply-To: <52157A3C.9010909@elyograg.org>
Subject: Re: How to avoid underscore sign indexing problem?
Date: Wed, 21 Aug 2013 23:15:44 -0400
MIME-Version: 1.0
Content-Type: text/plain;
	format=flowed;
	charset="iso-8859-1";
	reply-type=original
Content-Transfer-Encoding: 7bit
Importance: Normal
Sender: "Jack Krupansky" <jack@basetechnology.com>

"I thought that the StandardTokenizer always split on punctuation, "

Proving that you haven't read my book! The section on the standard tokenizer 
details the rules that the tokenizer uses (in addition to extensive 
examples.) That's what I mean by "deep dive."

-- Jack Krupansky

-----Original Message----- 
From: Shawn Heisey
Sent: Wednesday, August 21, 2013 10:41 PM
To: solr-user@lucene.apache.org
Subject: Re: How to avoid underscore sign indexing problem?

On 8/21/2013 7:54 PM, Floyd Wu wrote:
> When using StandardAnalyzer to tokenize string "Pacific_Rim" will get
>
> ST
> textraw_bytesstartendtypeposition
> pacific_rim[70 61 63 69 66 69 63 5f 72 69 6d]011<ALPHANUM>1
>
> How to make this string to be tokenized to these two tokens "Pacific",
> "Rim"?
> Set _ as stopword?
> Please kindly help on this.
> Many thanks.

Interesting.  I thought that the StandardTokenizer always split on
punctuation, but apparently that's not the case for the underscore
character.

You can always use the WordDelimeterFilter after the StandardTokenizer.

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.WordDelimiterFilterFactory

Thanks,
Shawn