Return-Path: 
 <java-user-return-28529-apmail-lucene-java-user-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-java-user-archive@www.apache.org
Received: (qmail 32582 invoked from network); 7 Jun 2007 21:07:43 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 7 Jun 2007 21:07:43 -0000
Received: (qmail 16984 invoked by uid 500); 7 Jun 2007 21:07:39 -0000
Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org
Received: (qmail 16956 invoked by uid 500); 7 Jun 2007 21:07:39 -0000
Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:java-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:java-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:java-user@lucene.apache.org>
List-Id: <java-user.lucene.apache.org>
Reply-To: java-user@lucene.apache.org
Delivered-To: mailing list java-user@lucene.apache.org
Received: (qmail 16945 invoked by uid 99); 7 Jun 2007 21:07:39 -0000
Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jun 2007 14:07:39 -0700
X-ASF-Spam-Status: No, hits=-0.0 required=10.0
	tests=SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (herse.apache.org: domain of DORONC@il.ibm.com designates
 195.212.29.155 as permitted sender)
Received: from [195.212.29.155] (HELO mtagate6.de.ibm.com) (195.212.29.155)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jun 2007 14:07:34 -0700
Received: from d12nrmr1607.megacenter.de.ibm.com
 (d12nrmr1607.megacenter.de.ibm.com [9.149.167.49])
	by mtagate6.de.ibm.com (8.13.8/8.13.8) with ESMTP id l57L7Ceh153868
	for <java-user@lucene.apache.org>; Thu, 7 Jun 2007 21:07:12 GMT
Received: from d12av02.megacenter.de.ibm.com (d12av02.megacenter.de.ibm.com
 [9.149.165.228])
	by d12nrmr1607.megacenter.de.ibm.com (8.13.8/8.13.8/NCO v8.3) with ESMTP id
 l57L7Cbe4165684
	for <java-user@lucene.apache.org>; Thu, 7 Jun 2007 23:07:12 +0200
Received: from d12av02.megacenter.de.ibm.com (loopback [127.0.0.1])
	by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id
 l57L7CuN024513
	for <java-user@lucene.apache.org>; Thu, 7 Jun 2007 23:07:12 +0200
Received: from d12mc102.megacenter.de.ibm.com (d12mc102.megacenter.de.ibm.com
 [9.149.167.114])
	by d12av02.megacenter.de.ibm.com (8.12.11.20060308/8.12.11) with ESMTP id
 l57L7CNt024506
	for <java-user@lucene.apache.org>; Thu, 7 Jun 2007 23:07:12 +0200
In-Reply-To: <46686AE4.8050808@curtin.com>
Subject: Re: np-pandock search problem (again, with more detail)
To: java-user@lucene.apache.org
X-Mailer: Lotus Notes Release 7.0 HF277 June 21, 2006
Message-ID: 
 <OFDFF02ED6.C306FBDB-ON882572F3.0071292B-882572F3.0073DCBD@il.ibm.com>
From: Doron Cohen <DORONC@il.ibm.com>
Date: Thu, 7 Jun 2007 14:05:31 -0700
X-MIMETrack: Serialize by Router on D12MC102/12/M/IBM(Release 7.0.2HF71 |
 November 3, 2006) at
 08/06/2007 00:07:11
MIME-Version: 1.0
Content-type: text/plain; charset=US-ASCII
X-Virus-Checked: Checked by ClamAV on apache.org

"Michael D. Curtin" <mike@curtin.com> wrote on 07/06/2007 13:30:28:

> > I think it splits by hyphens unless the no-hyphen
> > part has digits, so:
> >   np-pandock-a7
> > becomes
> >   np
> >   pandock-a7
> > This is for the indexing part.
>
> Wow!  Do you know the thinking behind that, i.e. why a number in a
> hyphenated expression prevents the split?

I actually asked myself the same question before the previous
post - javadocs for StandardAnalyzer just has the obvious - a
grammar-based tokenizer constructed with JavaCC.... - the wiki
page AnalysisParalysis also didn't explain much on the logic
behind it.

>From the StandardAnalyzer javacc grammar :
  // floating point, serial, model numbers, ip addresses, etc.
  // every other segment must have at least one digit
  <NUM: (<ALPHANUM> <P> <HAS_DIGIT> .... etc.
  <#P: ("_"|"-"|"/"|"."|",") >
My understanding of this: a non-whitespace sequence is broken
at either of these 5 chars
   _  -  /  .  ,
unless the part that follows part has a digit, in which case
it is assumed to be (part of) a serial no., model, etc.

Seems we can improve the documentation here.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org