lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doron Cohen (JIRA)" <j...@apache.org>
Subject [jira] Updated: (LUCENE-736) Sloppy Phrase Scoring Misbehavior
Date Sat, 02 Dec 2006 08:34:22 GMT
     [ http://issues.apache.org/jira/browse/LUCENE-736?page=all ]

Doron Cohen updated LUCENE-736:
-------------------------------

    Attachment: sloppy_phrase.patch2.txt
                res-search-orig2.log
                res-search-new2.log

The change to fix case 2 was not the main performance degradation cause.

I agree with Yonik that case 2 is much more important than case 1.
So I modified to fix case 2 but not case 1. 
Also extended the perf test to create also the "reversed" form of the sloppy phrases (slop
increased for reversed cases so that queries would match docs).

Cost of this fix dropped from 15% more CPU time to about 3%. 
I feel ok with this.

.....Operation..........runCnt...recsPerRun...rec/s..elapsedSec....avgUsedMem....avgTotalMem
Orig.SearchSameRdr_6000......4.........6000...194.2......123.59.....8,032,732.....11,333,632
New..SearchSameRdr_6000......4.........6000...187.5......128.02.....8,172,258.....11,333,632

Attached sloppy_phrase.patch2.txt has the updated fix, including both java and test parts.
Some of the asserts in the new tests were commented out cause the patch takes decision not
to fix case 1 above.

Also attaching the updates perf test logs - res-search-orig2.log and res-search-new2.log.

I did not compare scoring of similar cases between sloppy phrase and near spans and Paul suggested
- perhaps next week - not sure this should hold progress with this issue.

> Sloppy Phrase Scoring Misbehavior
> ---------------------------------
>
>                 Key: LUCENE-736
>                 URL: http://issues.apache.org/jira/browse/LUCENE-736
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: Search
>            Reporter: Doron Cohen
>         Assigned To: Doron Cohen
>            Priority: Minor
>         Attachments: perf-search-new.log, perf-search-orig.log, res-search-new2.log,
res-search-orig2.log, sloppy_phrase.patch2.txt, sloppy_phrase_java.patch.txt, sloppy_phrase_tests.patch.txt
>
>
> This is an extension of https://issues.apache.org/jira/browse/LUCENE-697
> In addition to abnormalities Yonik pointed out in 697, there seem to be other issues
with slopy phrase search and scoring.
> 1) A phrase with a repeated word would be detected in a document although it is not there.
> I.e. document = A B D C E , query = "B C B" would not find this document (as expected),
but query "B C B"~2 would find it. 
> I think that no matter how large the slop is, this document should not be a match.
> 2) A document containing both orders of a query, symmetrically, would score differently
for the queru and for its reveresed form.
> I.e. document = A B C B A would score differently for queries "B C"~2 and "C B"~2, although
it is symmetric to both.
> I will attach test cases that show both these problems and the one reported by Yonik
in 697. 

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message