lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Miller (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-2878) Allow Scorer to expose positions and payloads aka. nuke spans
Date Sat, 22 Jan 2011 14:53:43 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985125#action_12985125
] 

Mark Miller commented on LUCENE-2878:
-------------------------------------

bq. I don't like PayloadSpanUtil either, i think its a huge trap and it should move to src/test
or be removed.

It's not that bad of a trap :) I has warnings all over it - some people use it out there.
I think it's more useful than trapful - if not just as demo code.

It is src/test quality at best though though. I never would have committed it to core - I
expected to put it in contrib at best. But someone else committed it at the time, so I just
did a 'meh' and put warnings about experimental on it. I forgot to take it out of the patch
on the issue or something. I never even used it or tested it much myself - I just wanted to
put in some example/test code for this use case - I know some users on the list have used
it for some form of simple highlighting, where the coords to highlight are in the payloads.
I wouldn't trust it these days though :)

It will be found less if its in src/tests, but that is probably a fine place for it - it would
need some love/support above what it has to really make sense in any module/contrib. I've
never really meant to maintain it myself - and I've never really felt it belonged in core
(just never had the itch to squash it either).

> Allow Scorer to expose positions and payloads aka. nuke spans 
> --------------------------------------------------------------
>
>                 Key: LUCENE-2878
>                 URL: https://issues.apache.org/jira/browse/LUCENE-2878
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>    Affects Versions: Bulk Postings branch
>            Reporter: Simon Willnauer
>            Assignee: Simon Willnauer
>         Attachments: LUCENE-2878.patch
>
>
> Currently we have two somewhat separate types of queries, the one which can make use
of positions (mainly spans) and payloads (spans). Yet Span*Query doesn't really do scoring
comparable to what other queries do and at the end of the day they are duplicating lot of
code all over lucene. Span*Queries are also limited to other Span*Query instances such that
you can not use a TermQuery or a BooleanQuery with SpanNear or anthing like that. 
> Beside of the Span*Query limitation other queries lacking a quiet interesting feature
since they can not score based on term proximity since scores doesn't expose any positional
information. All those problems bugged me for a while now so I stared working on that using
the bulkpostings API. I would have done that first cut on trunk but TermScorer is working
on BlockReader that do not expose positions while the one in this branch does. I started adding
a new Positions class which users can pull from a scorer, to prevent unnecessary positions
enums I added ScorerContext#needsPositions and eventually Scorere#needsPayloads to create
the corresponding enum on demand. Yet, currently only TermQuery / TermScorer implements this
API and other simply return null instead. 
> To show that the API really works and our BulkPostings work fine too with positions I
cut over TermSpanQuery to use a TermScorer under the hood and nuked TermSpans entirely. A
nice sideeffect of this was that the Position BulkReading implementation got some exercise
which now :) work all with positions while Payloads for bulkreading are kind of experimental
in the patch and those only work with Standard codec. 
> So all spans now work on top of TermScorer ( I truly hate spans since today ) including
the ones that need Payloads (StandardCodec ONLY)!!  I didn't bother to implement the other
codecs yet since I want to get feedback on the API and on this first cut before I go one with
it. I will upload the corresponding patch in a minute. 
> I also had to cut over SpanQuery.getSpans(IR) to SpanQuery.getSpans(AtomicReaderContext)
which I should probably do on trunk first but after that pain today I need a break first :).
> The patch passes all core tests (org.apache.lucene.search.highlight.HighlighterTest still
fails but I didn't look into the MemoryIndex BulkPostings API yet)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message