Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
Date: Wed, 4 Jul 2007 14:05:17 -0400 (EDT)
From: Tim Sturge <tsturge@metaweb.com>
To: java-user@lucene.apache.org
Cc: hossman_lucene@fucit.org
Message-ID: <20153459.717571183572317952.JavaMail.root@h00215>
In-Reply-To: <30247405.717551183571286058.JavaMail.root@h00215>
Subject: Re: product based term combination for BooleanQuery?
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


:-) The use of wikipedia data here is no secret; it's all over www.freebase.com. I just hoped to avoid being sucked into a "what is the best way to index wikipedia with Lucene?" discussion, which I believe several other groups are already tackling.

At index time, I used a per document boost (over all fields) and a per field bost (over all documents). I can certainly factor out the first into a query boost, but I was under the impression that if I ever wanted to combine fields (eg to index all "name" "alias" and "title" data in a single "head" field) then I had to pre-boost the data prior to combining it. I tend to believe that these (short) fields contain more relevant information than (long) wikipedia articles or other documents. 

Should idf and tf take care of that short/long quality distinction? It sounds like you feel they should. 
I'll build an index without the per field boost and see if that produces improved results.

Thanks,

Tim

----- Original Message -----
From: "Chris Hostetter" <hossman_lucene@fucit.org>
To: "Lucene Users" <java-user@lucene.apache.org>
Sent: Tuesday, July 3, 2007 10:26:57 PM (GMT-0800) America/Los_Angeles
Subject: Re: product based term combination for BooleanQuery?


(side note: if you are going to try and obfuscate your field names when
sending explain output so we don't know you are using wikipedia data (not
that we care), please at least be consistent about it so the final
explanations actual make sense -- it will save everyone a lot of confusion
and help us help you)

the biggest factor in your scores seems to be the fieldNorms for your
name, title and alias fields ... they are so high, that tf and idf are
pretty much irrelevant.

By the looks of it, when you were indexing your docs, you used a
consistent field boost per field on every instance of that field for every
document ... this is really not a use case where index time field (or
document) boosts make sense.  in my opinion hte number one thing you can
do to imrpove your relevency right now is to stop using index time
boosts and use query boosts instead.

If you don't want to reindex completely the LengthNormModifier class (in
the misc contrib) can update all of your norms in place without reindexing
and throw away any index time boosts you had.


-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org