Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
MIME-Version: 1.0
In-Reply-To: <CAOdYfZXD+4fYSskWZY7dxiq1EKZfh9tv_PpcRVhiwXF8+qa-8w@mail.gmail.com>
References: <CAB1V9m5Ze=vFCTV721kyzz9p3ODQf7nR_DRDA1jymq6ZCfEh7g@mail.gmail.com>
 <CAOdYfZXD+4fYSskWZY7dxiq1EKZfh9tv_PpcRVhiwXF8+qa-8w@mail.gmail.com>
From: Chitra <chithu.r111@gmail.com>
Date: Fri, 20 Oct 2017 14:04:17 +0530
Message-ID: <CAB1V9m6STk_1QmkJqWtwhnJFfefyEn9M7ee1D46zeeqF8OX-VA@mail.gmail.com>
Subject: Re: ClassicAnalyzer Behavior on accent character
To: Lucene Users <java-user@lucene.apache.org>
Content-Type: multipart/alternative; boundary="001a1142ec8cda6807055bf6579f"
archived-at: Fri, 20 Oct 2017 08:34:47 -0000

--001a1142ec8cda6807055bf6579f
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Hi Robert,
                 Yes, standardTokenizer solves my case... could you please
explain the difference between ClassicalTokenizer and StandardTokenizer?
How does standardTokenizer solve my case? I surf the web but I was unable
to understand...


Any help is greatly appreciated.

On Fri, Oct 20, 2017 at 12:10 AM, Robert Muir <rcmuir@gmail.com> wrote:

> easy, don't use classictokenizer: use standardtokenizer instead.
>
> On Thu, Oct 19, 2017 at 9:37 AM, Chitra <chithu.r111@gmail.com> wrote:
> > Hi,
> >               I indexed a term '=E2=92=B6e=C5=98=EA=9D=8B=EA=9D=92=C9=
=AB=E2=B1=AF=C5=8B=C9=87' (aeroplane) and the term was
> > indexed as "er l n", some characters were trimmed while indexing.
> >
> > Here is my code
> >
> > protected Analyzer.TokenStreamComponents createComponents(final String
> >> fieldName, final Reader reader)
> >>     {
> >>         final ClassicTokenizer src =3D new ClassicTokenizer(getVersion=
(),
> >> reader);
> >>         src.setMaxTokenLength(ClassicAnalyzer.DEFAULT_MAX_
> TOKEN_LENGTH);
> >>
> >>         TokenStream tok =3D new ClassicFilter(src);
> >>         tok =3D new LowerCaseFilter(getVersion(), tok);
> >>         tok =3D new StopFilter(getVersion(), tok, stopwords);
> >>         tok =3D new ASCIIFoldingFilter(tok); // to enable
> AccentInsensitive
> >> search
> >>
> >>         return new Analyzer.TokenStreamComponents(src, tok)
> >>         {
> >>             @Override
> >>             protected void setReader(final Reader reader) throws
> >> IOException
> >>             {
> >>
> >> src.setMaxTokenLength(ClassicAnalyzer.DEFAULT_MAX_TOKEN_LENGTH);
> >>                 super.setReader(reader);
> >>             }
> >>         };
> >>     }
> >
> >
> >
> > Am I missing anything? Is that expected behavior for my input or any
> reason
> > behind such abnormal behavior?
> >
> > --
> > Regards,
> > Chitra
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


--=20
Regards,
Chitra

--001a1142ec8cda6807055bf6579f--