lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Busch <>
Subject Re: Payloads
Date Wed, 20 Dec 2006 19:42:02 GMT
Doug Cutting wrote:
> Michael,
> This sounds like very good work.  The back-compatibility of this 
> approach is great.  But we should also consider this in the broader 
> context of index-format flexibility.
> Three general approaches have been proposed.  They are not exclusive.
> 1. Make the index format extensible by adding user-implementable 
> reader and writer interfaces for postings.
> 2. Add a richer set of standard index formats, including things like 
> compressed fields, no-positions, per-position weights, etc.
> 3. Provide hooks for including arbitrary binary data.
> Your proposal is of type (3).  LUCENE-662 is a (1).  Approaches of 
> type (2) are most friendly to non-Java implementations, since the 
> semantics of each variation are well-defined.
> I don't see a reason not to pursue all three, but in a coordinated 
> manner.  In particular, we don't want to add a feature of type (3) 
> that would make it harder to add type (1) APIs.  It would thus be best 
> if we had a rough specification of type (1) and type (2).  A proposal 
> of type (2) is at:
> But I'm not sure that we yet have any proposed designs for an 
> extensible posting API.  (Is anyone aware of one?)  This payload 
> proposal can probably be easily incorporated into such a design, but I 
> would have more confidence if we had one.  I guess I should attempt one!


thanks for your detailed response. I'm aware that the long-term goal is 
the flexible index format and I see the payloads patch only as a part of 
it. The patch focuses on extending the index data structures and about a 
possible payload encoding. It doesn't focus yet on a flexible API, it 
only offers the two mentioned low-level methods to add and retrieve byte 

I would love to work with you guys on the flexible index format and to 
combine my patch with your suggestions and the patch from Nicolas! I 
will look at your proposal and Nicolas' patch tomorrow (have to go now). 
I just attached my patch (LUCENE-755), so if you get a chance you could 
take a look at it.

Maybe it would make sense now to follow your suggestion you made earlier 
this year and start a new package to work on the new index format? On 
the other hand, if people would like to use the payloads soon I guess 
due to the backwards compatibility it would be low risk to add it to the 
current index format to provide this feature until we can finish the 
flexible format?

- Michael

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message