Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of ian.lea@gmail.com designates
 209.85.210.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1319818396270-3461423.post@n3.nabble.com>
References: <1319818396270-3461423.post@n3.nabble.com>
From: Ian Lea <ian.lea@gmail.com>
Date: Fri, 28 Oct 2011 18:05:00 +0100
Message-ID: 
 <CAEY5pxVQofYk91q_W69xXWbZVYyp97155cSXiz-5a1z4+vtS5w@mail.gmail.com>
Subject: Re: multiple phrase search for topic
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Seems to me your approach should work, although I'd worry about performance=
.

> A lot of top-ranked documents are not the best candidates for the "Softwa=
re Technology" topic, even
> though they contain the phrases (not very frequent)

Surely the docs that contain the phrases are going to be top of the
list?  In what way are others "better" than the ones ranked top?

Running queries with a large number of clauses on large indexes can be
slow.  I'd look into doing the categorisation at indexing time then
searching with a simple "category: Software Technology" clause.  Or
filter.

Projects such as Carrot2 or LingPipe may be worth a look.


--
Ian.


On Fri, Oct 28, 2011 at 5:13 PM, deb.lucene <deb.lucene@gmail.com> wrote:
> Hi Group,
>
> I am indexing and searching a large corpus of news articles. The indexing
> process is very straightforward, I am utilizing the standardAnalyzer and
> analyzing the content of the news document.
> **************************
> document =3D new Document();
> document.add(new Field("snum", snum, Field.Store.YES,Field.Index.NO));
> document.add(new Field("content", conent,
> Field.Store.NO,Field.Index.ANALYZED,Field.TermVector.YES));
> indexWriter.addDocument(document);
>
> where, "snum" is the serial number of the news article and "content" is t=
he
> actual text of the document.
>
> ******************************
> So far so good. The searching process is little complex as I am doing a
> multiple phrase searching. Let me explain the situation with an example.
> Suppose I have to retrieve documents which belong to the category "Softwa=
re
> Technology" using phrase/query terms related to that topic. Also, I have
> around 10k phrases which belong to this particular category (e.g. "data
> recovery tool",....., "C++ language",...."Steve Jobs",....."Mac
> Layer",...."Grid Computing"...etc.). My idea was to create separate phras=
e
> query for each of these phrases and then add all of them to a boolean que=
ry.
> Much like this,
>
> ****************************
> PhraseQuery pQuery ;
> BooleanQuery bQuery =3D new BooleanQuery ();
> bQuery.setMaxClauseCount(10000);
>
> for (Phrase phrase : allPhrases)
> {
> =A0 =A0 =A0 =A0 =A0String terms[] =3D phrase.split("\\s++");
> =A0 =A0 =A0 =A0 =A0int words =3D terms.length ;
>
> =A0 =A0 =A0 =A0 =A0pQuery =3D new PhraseQuery();
> =A0 =A0 =A0 =A0 =A0for ( int j =3D 0 ; j < words ; j++)
> =A0 =A0 =A0 =A0 =A0 {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 String word =3D terms[j].toLowerCase();
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 pQuery.add(new Term("content", word));
>
> =A0 =A0 =A0 =A0 =A0 }
> =A0 =A0 =A0 =A0 =A0 pQuery.setSlop(0);
> =A0 =A0 =A0 =A0 =A0 bQuery.add(pQuery,BooleanClause.Occur.SHOULD);
> }
> int numOfSugg =3D 2000 ;
> TopDocs matches =3D isearcher.search(bQuery, numOfSugg)
>
> ********************************
> Unfortunately when I am searching the news content with this approach the
> searched results do not look very promising. A lot of top-ranked document=
s
> are not the best candidates for the "Software Technology" topic, even tho=
ugh
> they contain the phrases (not very frequent). My questions are :
>
> 1) is there anything wrong in this usage of the phrase/boolean query?
> 2) how I can guarantee to retrieve the most suitable news documents (i.e.
> document which contains a lot of the related phrases) in the top searched
> results? I utilized the BooleanClause.Occur.SHOULD feature (instead of th=
e
> MUST) because it is impossible to find a single document containing all o=
f
> the 10k phrases, but using the SHOULD feature I surmise the best results
> will be which contains at least a few of the phrases.
>
> thanks in advance,
> --d
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/multiple=
-phrase-search-for-topic-tp3461423p3461423.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org