Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of ralf.bierig@gmail.com
 designates 209.85.212.173 as permitted sender)
Message-ID: <544F5CBC.6070600@gmail.com>
Date: Tue, 28 Oct 2014 10:07:08 +0100
From: Ralf Bierig <ralf.bierig@gmail.com>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:24.0) Gecko/20100101 Thunderbird/24.6.0
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: Indexing Weighted Tags per Document
References: 
 <CA+40OK+-37ecM45jtZGWDDbj4DAk-iv9reJfpZ1toK=4MqQ8rw@mail.gmail.com>
	<1414420856937-4166094.post@n3.nabble.com>
	<034c01cff1f6$276c6de0$764549a0$@efendi.ca>
	<1414424827939-4166116.post@n3.nabble.com>	<544E6A1C.9050605@gmail.com>
	<544E6DDF.6050109@gmail.com>
 <CADMUOEVC6gBEXs9EP-FsoV7u+mHEpyS=0oWL+_k6N-LOkB9DAQ@mail.gmail.com>
In-Reply-To: 
 <CADMUOEVC6gBEXs9EP-FsoV7u+mHEpyS=0oWL+_k6N-LOkB9DAQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

The second solution sounds great and a lot more natural than payloads.

I know how to overwrite the Similarity class but this one would only be 
called at search time and then already use the existing term frequency. 
Looking up the probabilities every time a search is performed is 
probably also not performing well. So, I suspect I would somehow need to 
find a way to store the term frequency directly into the index at the 
time when I am indexing documents. Is that correct?

Do you have a code sniplet that would highlight that part of your 
elegant solution?

Thanks in advance,
Ralf

On 28.10.2014 09:31, Ramkumar R. Aiyengar wrote:
> There are a few approaches possible here, we had a similar use case and
> went for the second one below. I primarily deal with Solr, so I don't know
> of Lucene-only examples, but hopefully you can dig this up..
>
> (1) You can attach payloads to each occurrence of the tag, and modify the
> scoring to use the payload..
>
> (2) Use term frequency as a proxy. You could scale the probability by a
> factor and introduce the term as many times as the scaled value
> (essentially making it the term frequency). Scoring will know account for
> this. Advantage is that you also can achieve score normalisation with
> keywords and amongst tags, and you can also filter results by probability.
>
> (3) There potentially is also a solution using child documents and block
> join, but I may be mistaken, haven't explored this a lot..
>   On 27 Oct 2014 16:10, "Ralf Bierig" <ralf.bierig@gmail.com> wrote:
>
>> I want to index documents together with a list of tags (usually between
>> 10-30) that represent meta information about this document. Normally, i
>> would create an extra field "tag" store every tag, by its name, inside that
>> field and create my 10-30 fields that and adding it to the document before
>> adding the document to the index and writing the index.
>>
>> However, I have the following extra requirements:
>>
>> a) I need to have a weight in the range of [0,1] being associated with the
>> tag that represents the probability of this tag being true.
>>
>> b) These tags must be associated with the document and not with the terms
>> of the document.
>>
>> c) I must be able to associate many tags to a document instance.
>>
>> d) I must be able to use the weight in the weighting process of the search
>> engine.
>>
>> e) The weight must be for the document instance, as the weight represents
>> the probability for that tag for that particular document. E.g.
>>
>> fieldname: tag
>> fieldvalue: tree
>> fieldweight: 0.8
>>
>> meaning that this particular document is with a probability of 0.8 about
>> trees.
>>
>> What is the best way to do that?
>> Can somebody point me to an example or something quite similar that
>> captures such a problem?
>>
>> Best,
>> Ralf
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org