lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <>
Subject Re: [jira] Updated: (LUCENE-755) Payloads
Date Mon, 12 Mar 2007 12:38:20 GMT
I haven't looked at your latest patch yet, so this is just guesswork,  
but was thinking in TermScorer, around line 75 or so, we could add:

score *= similarity.scorePayload(payloadBuffer);

The default Similarity would just return 1.  This would allow people  
to incorporate a score based on what is in the payload, per their  
application needs and would be completely backward-compatible.  We  
may even want to postpone the decoding of the payload to inside the  
Similarity for performance reasons, but that should be tested, since  
that could be cause for confusion for people overriding Similarity.   
I will have to look at some of the other Scorers to see if there is a  
way to incorporate into some of them.

None of this would prevent using payloads for other things as well,  
such as the XPath query example.

Doing this would involve switching over to using TermPositions like  
we talked about.  Like I said, I will take a look at it and see if  
anything resonates.


On Mar 11, 2007, at 11:26 PM, Michael Busch wrote:

> Grant Ingersoll wrote:
>> Cool.  I will try and take a look at it tomorrow.  Since we have  
>> the lazy SegTermPos thing in now, we should be able to integrate  
>> this into scoring via the Similarity and merge TermDocs and  
>> TermPositions like you suggested.
>> If I can get the Scoring piece in and people are fine w/ the  
>> flushBuffer change then hopefully we can get this in this week.  I  
>> will try to post a patch that includes your patch and the scoring  
>> integration by tomorrow or Tuesday if that is fine with you.
> I'm not completely sure how you want to integrate this in the  
> Similarity class. Payloads can not only be used for scoring.  
> Consider for example XML search: the payloads can be used here to  
> store in which element a term occurs. During search (e. g. an XPath  
> query) the payloads would be used then to find hits, not for scoring.
> On the other hand if you want to store e. g. per-postions boosts in  
> the payloads, you could use the norm en/decoding methods that are  
> already in Similarity. You could use the following code in a  
> TokenStream:
>  byte[] payload = new byte[1];
>  payload[0] = Similari.encodeNorm(boost);
>  token.setPayload(payload);
> and in a scorer you could get the boost then with:
>  termPositions.getPayload(payloadBuffer);
>  float boost = Similarity.decodeNorm(payloadBuffer[0]);
> But maybe you have something different in mind? Could you  
> elaborate, please?
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

Grant Ingersoll
Center for Natural Language Processing

Read the Lucene Java FAQ at 

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message