Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of phil123@gmail.com designates
 209.85.211.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=gkPP05xccFVThQtc9DOeEqXVP33N6LSMeitOT0TKkOAoxma2zdkquvg2IobpGQikcT
         beitsbPRVq5b/cfCo/GxenPMSGlsj1DL3acCi83RF8+mA8iaNunrwq83k4li1MVuHs0l
         RtBBbM3wY4cDTNUosDTqZc/QWB8q2jit68ROQ=
MIME-Version: 1.0
In-Reply-To: <786fde50908040356v15db635buf3063d5b7a45a5f1@mail.gmail.com>
References: <24802552.post@talk.nabble.com>
	 <867513fe0908032320m61fd55f7qa6b40ca9c625a343@mail.gmail.com>
	 <24803073.post@talk.nabble.com>
	 <786fde50908032354m656b2260u6f1fa44feee1987c@mail.gmail.com>
	 <24803560.post@talk.nabble.com>
	 <786fde50908040128p67c2ca6en3c23ad7550280a1b@mail.gmail.com>
	 <24805609.post@talk.nabble.com>
	 <786fde50908040356v15db635buf3063d5b7a45a5f1@mail.gmail.com>
Date: Tue, 4 Aug 2009 08:19:06 -0700
Message-ID: <9cafbc680908040819v17d76a27u7c256b7065c815bf@mail.gmail.com>
Subject: Re: Searching doubt
From: Phil Whelan <phil123@gmail.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On Tue, Aug 4, 2009 at 3:56 AM, Shai Erera<serera@gmail.com> wrote:
> 2) Use a dictionary (real dictionary), and search it for every substring,
> e.g. "a", "ab", "abo" ... "about" etc. If you find a match, split it there.
> This needs some fine tuning, like checking if the rest is also a word and if
> the full string is also a word, so that you don't break up meaningful words.
> You'll need to get a dictionary for that.

I do not have a solution to this, but it strikes me as very similar to
they way you traverse Japanese to break words, since that has no
spaces. Is there a Japanese tokenizer and, if so, does it handle this?
If so, you could replace the Japanese dictionary with an English
dictionary. Just a random thought had that might / might not help.

Phil

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org