Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of rooseve6891@gmail.com
 designates 209.85.216.204 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=V4xTRMtCRelj+SBb4YLTVXMufjEG4M2szH7raGBxRlUpVKH6MOR/ZvgjTqFSCO5v5Z
         NYjCxYlx2wj+qKhpTEiXrSjNl4Ju8u8r+2YpTLB3hsxyRsFYUv0gekPZnOmaHFOdyPVs
         K0EMQqTJfK8Xc/G9XkUycf4aif4BqD96xQhb0=
MIME-Version: 1.0
Date: Tue, 6 Oct 2009 19:42:23 +0800
Message-ID: <8680aede0910060442o5a0b6058rc30139af13b90265@mail.gmail.com>
Subject: Phase Extraction, mainly for English
From: Andrew Zhang <rooseve6891@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0016e6477a943ea6c9047542b907

--0016e6477a943ea6c9047542b907
Content-Type: text/plain; charset=ISO-8859-1

Hi guys,

The requirement is very simple here, e.g. for this sentence, 'The NBA
formally announced its new *social media* guidelines Wednesday',  I want to
treat '*social media*' as a whole phase term. The default english analyzers
came with lucene all deal with single word, so it you want to get the most
frequent terms, *social *and *media* are separated, and each of them can't
represent a good meaning as *social media*, right?

I know there's a way built on some phase dictionary, and try to match the
phase already there, very like the way to do with chinese language, but is
there an open source solution for english, I mean I don't want to build a
phase dictionary myself, and I also want a smart way, which can "discover"
the phase automatically. I got 2 millions docs analyzered the norma way, all
single terms, which I can use as a base source, and it's possible to find
that *social media *came together frequently, but I really don't know what's
the reverse way.

I tried to find some phase analyzers, but no luck. so any advices?

Regards,
Andrew
-- 
Simple is best

--0016e6477a943ea6c9047542b907--