Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of rcmuir@gmail.com designates
 209.85.212.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALGBXaM3jSBBB-PkPBykrWW8ATY3Sq=xiCsxfZpG0eV8qntw_w@mail.gmail.com>
References: 
 <CALGBXaM3jSBBB-PkPBykrWW8ATY3Sq=xiCsxfZpG0eV8qntw_w@mail.gmail.com>
From: Robert Muir <rcmuir@gmail.com>
Date: Sun, 6 May 2012 19:32:08 -0400
Message-ID: 
 <CAOdYfZXrswB5AwdLfnJHjcBNT=HMGdHQd5YXM=sY8k4wFxqzBg@mail.gmail.com>
Subject: Re: Calculating IDF value more efficiently
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Look at IndexReader.docFreq

On Fri, Apr 27, 2012 at 10:38 PM, Kasun Perera <kasunp@opensource.lk> wrote=
:
> This is my program to calculate TF-IDF value for a document in a collecti=
on
> of documents. This is working fine, but takes lot of time when calculatin=
g
> the "IDF" values (finding the no of documents which contains particular
> term).
>
> Is there a more efficient way of finding the no of documents which contai=
ns
> a particular term?
>
> freq =3D termsFreq.getTermFrequencies();
>
> terms =3D termsFreq.getTerms();
>
> int noOfTerms =3D terms.length;
>
> score =3D new float[noOfTerms];
> DefaultSimilarity simi =3D new DefaultSimilarity();
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0for (i =3D 0; i < noOfTerms; i++) {
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0int noofDocsContainTerm =3D noOf=
DocsContainTerm(terms[i]);
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0float tf =3D simi.tf(freq[i]);
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0float idf =3D simi.idf(noofDocsC=
ontainTerm, noOfDocs);
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0score[i] =3D tf * idf ;
>
> =C2=A0 =C2=A0 =C2=A0 =C2=A0}
>
> ////
>
> public int noOfDocsContainTerm(String querystr) throws
> CorruptIndexException, IOException, ParseException{
>
> QueryParser qp=3Dnew QueryParser(Version.LUCENE_35, "docuemnt", new
> StandardAnalyzer(Version.LUCENE_35));
>
> Query q=3Dqp.parse(querystr);
>
> int hitsPerPage =3D docNames.length; //minumum number or search results
> IndexSearcher searcher =3D new IndexSearcher(ramMemDir, true);
> TopScoreDocCollector collector =3D TopScoreDocCollector.create(hitsPerPag=
e, true);
>
> searcher.search(q, collector);
>
> ScoreDoc[] hits =3D collector.topDocs().scoreDocs;
>
> =C2=A0 =C2=A0return hits.length;
> }
>
>
> --
> Regards
>
> Kasun Perera


--=20
lucidimagination.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org