lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kent Gibson <kentgib...@yahoo.com>
Subject Re: raw hit count
Date Sun, 30 Nov 2003 16:13:56 GMT
Thanks for the help guys, but unfortunately I am still
stuck. Let me reiterate what I would like to do and
then explain what I have tried.

I would like to know that in document x the query y
appeared n times. 

For example:

query = "Bank" : Bank found in doc number 1, 3 tims

Understandbly this is a bit tricky when query y is
composed of more than one word, but for the moment I
would be satisified if I knew how many times query y
appeared in its entirety.

However in the end it would be great if I could get a
result as follows: 
query = "Hells Bells"; Hells found in doc number 2, 3
times and Bells Found 0 times 

as per Erik's idea I tried with the BitSet as follows:

QueryFilter qf = new QueryFilter(query);
IndexReader ir = IndexReader.open(indexPath);
Searcher searcher2 = new IndexSearcher(ir);

// get the bit set for the query
BitSet bits = qf.bits(ir);
last = bits.nextSetBit(offset);
offset = last + 1;

System.out.println("First bit is: " + last);	
System.out.println("Bits " + bits.toString());
		
// clear all the bits
bits.clear();
System.out.println("Bits after " + bits.toString());
bits.set(last);
		
/* just to see the effect */	
BitSet bits2 = qf.bits(ir);
System.out.println("Bits now " + bits2.toString());
		
Hits hits2 = searcher2.search(query,qf);
/* this value is always one /*
  */	
System.out.println("raw hits : " + hits2.length());

However I always get a result of 1, which I suppose is
has to do with this overlap thingy.

As per Ype's idea I tried to implement a Similarity
object, but two things I believe are wrong, a) I am
doing something fundamentally wrong with the maths b)
I get a sneaky idea this is the wrong way around this.

Is there not a simple way to just get some word
statistics out of a file?

Once again thanks for the inputs and I look forward to
a long fight.

public float lengthNorm(String fieldName, int
numTerms)
{
return (float) 1.0 ;
}

/** Implemented as <code>sqrt(freq)</code>. */
public float tf(float freq)
{
return (float) (freq);
}

/** Implemented as <code>log(numDocs/(docFreq+1)) +
1</code>. */
public float idf(int docFreq, int numDocs)
{
return (float)1.0;

}
--- Ype Kingma <ykingma@xs4all.nl> wrote:
> Kent, Erik,
> 
> On Saturday 29 November 2003 17:20, Erik Hatcher
> wrote:
> > I enjoy at least attempting to answer questions
> here, even if I'm half
> > wrong, so by all means correct me if I
> misspeak....
> 
> Me too, :)
> 
> > On Saturday, November 29, 2003, at 06:37  PM, Kent
> Gibson wrote:
> > > All I would like to know is how many times a
> query was
> > > found in a particular document. I have no
> problems
> > > getting the score from hits.score(). hits.length
> is
> > > the number of times in total that the query was
> found,
> > > however I want the the number of times the query
> was
> > > found on a document by document basis. is this
> > > possible?
> 
> Could you be a bit more precise on what you mean
> by 'the number of times the query was found'? For a
> single
> query term, it is straightforward, but what about
> eg. a query for three
> optional terms?
> 
> >
> > The 'coord' factor used in computing the score is
> exactly this.  See
> > the javadoc for it:
> >
> > 
>
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/search/
> > Similarity.html#coord(int,%20int)
> 
> AFAIK, this overlap is the number of terms the
> document and the query
> have in common.
> For a query consisting of a single term, the overlap
> is always one,
> and the number of times the query occurs in a
> document is the term frequency
> in the document.
> 
> > You could implement a custom Similarity to capture
> the "overlap" or
> > adjust the the factor depending on what you're
> trying to accomplish.
> >
> > >  The only idea I have is to rerun the search,
> > > but I can't even see how to run a search on only
> one
> > > document!
> >
> > You could always rerun a search with a Filter with
> only one bit enabled
> > and see if zero or one document is returned - that
> would be quite
> > trivial and fast.
> 
> You could also implement a Similarity that ignores
> the total number
> of terms in the searched document field, see
> lengthNorm() in
>
http://jakarta.apache.org/lucene/docs/api/org/apache/lucene/search/Similarity.html
> As lengthNorm() is applied at indexing time, you
> will have to reindex
> for this to work for you.
> At query time you can then use a tf() implementation
> that is linear, instead
> of the default square root in DefaultSimilarity, and
> a constant idf(),
> instead of the default log of the inverse document
> frequency.
> You should then get a document score that is
> proportional
> to the number of query terms in the document.
> 
> Kind regards,
> Ype
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 


__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message