Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of vcomandur@gmail.com designates
 209.85.216.204 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Q3hO6nXCq0m6+FdpdE/s1MXKjZVWLk240c3evFOBJ0N+fEMNOVs5p81TdVg1VAlCDP
         uRWcWbk5kNNBV+7qTRb2ItigerARnTr/Ro4c/Zy13zRHj2uw975f/hGNbQRvCpQWCnhJ
         4rMSVKD4N461HLMDuTz20LARr9RHIvRl7CVa8=
MIME-Version: 1.0
In-Reply-To: <8680aede0910060442o5a0b6058rc30139af13b90265@mail.gmail.com>
References: <8680aede0910060442o5a0b6058rc30139af13b90265@mail.gmail.com>
Date: Tue, 6 Oct 2009 17:40:54 +0530
Message-ID: <a59386fa0910060510p5f1ff643t4a36255e3127cb16@mail.gmail.com>
Subject: Re: Phase Extraction, mainly for English
From: Vasudevan Comandur <vcomandur@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0016368e270443160d0475431fb5

--0016368e270443160d0475431fb5
Content-Type: text/plain; charset=ISO-8859-1

Hi,

   Take the NLP route and use modules like POS tagger and NP chunker.

   OpenNLP has a stack for English language. Try to use them.

Regards
 Vasu

On Tue, Oct 6, 2009 at 5:12 PM, Andrew Zhang <rooseve6891@gmail.com> wrote:

> Hi guys,
>
> The requirement is very simple here, e.g. for this sentence, 'The NBA
> formally announced its new *social media* guidelines Wednesday',  I want to
> treat '*social media*' as a whole phase term. The default english analyzers
> came with lucene all deal with single word, so it you want to get the most
> frequent terms, *social *and *media* are separated, and each of them can't
> represent a good meaning as *social media*, right?
>
> I know there's a way built on some phase dictionary, and try to match the
> phase already there, very like the way to do with chinese language, but is
> there an open source solution for english, I mean I don't want to build a
> phase dictionary myself, and I also want a smart way, which can "discover"
> the phase automatically. I got 2 millions docs analyzered the norma way,
> all
> single terms, which I can use as a base source, and it's possible to find
> that *social media *came together frequently, but I really don't know
> what's
> the reverse way.
>
> I tried to find some phase analyzers, but no luck. so any advices?
>
> Regards,
> Andrew
> --
> Simple is best
>

--0016368e270443160d0475431fb5--