lucene-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris M. Hostetter (Jira)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-13946) SpellCheckCollatorTest.testEstimatedHitCounts reproducing failure seed
Date Tue, 19 Nov 2019 22:20:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-13946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977881#comment-16977881
] 

Chris M. Hostetter commented on SOLR-13946:
-------------------------------------------

Background...

Jenkins recently identified a reproducing seed failure for SpellCheckCollatorTest.testEstimatedHitCounts
on branch_8_3 that reproduces on branch_8x...
{noformat}
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SpellCheckCollatorTest -Dtests.method=testEstimatedHitCounts
-Dtests.seed=AFA731DEE618DA14 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
-Dtests.badapples=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-8.3/test-data/enwiki.random.lines.txt
-Dtests.locale=ga-IE -Dtests.timezone=Canada/Atlantic -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.16s J2 | SpellCheckCollatorTest.testEstimatedHitCounts <<<
   [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
   [junit4]    >        at __randomizedtesting.SeedInfo.seed([AFA731DEE618DA14:9E1C8FEB4327CAC4]:0)
   [junit4]    >        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:998)
   [junit4]    >        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:958)
   [junit4]    >        at org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:569)
   [junit4]    >        at java.lang.Thread.run(Thread.java:748)
   [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/long[@name='hits'
and 3 <= . and . <= 13]
   [junit4]    >        xml response was: <?xml version="1.0" encoding="UTF-8"?>
   [junit4]    > <response>
   [junit4]    > <lst name="responseHeader"><int name="status">0</int><int
name="QTime">2</int></lst><result name="response" numFound="0" start="0"></result><lst
name="spellcheck"><lst name="suggestions"><lst name="everother"><int name="numFound">1</int><int
name="startOffset">9</int><int name="endOffset">18</int><arr name="suggestion"><str>everyother</str></arr></lst></lst><lst
name="collations"><lst name="collation"><str name="collationQuery">teststop:everyother</str><long
name="hits">14</long><lst name="misspellingsAndCorrections"><str name="everother">everyother</str></lst></lst></lst></lst>
   [junit4]    > </response>
   [junit4]    >        request was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=/spellCheckCompRH&q=teststop:everother&spellcheck.collateMaxCollectDocs=5
   [junit4]    >        at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:991)
   [junit4]    >        ... 41 more
{noformat}
...that seed did not reproduce on branch_8_0, so i attempted to use "git bisect" to idenfity
when exactly it started to fail, but that lead to a false end (651f41e21bd3df98f70d2673295db29506e3d2e6)
where a new variable was introduced in RandomCodec skewing the results. For completeness,
i manually tried overriding the random FSTLoadMode in RandomCodec, after consuming hte same
amount of randomness, and got a failure from this seed for all possible values – supporting
the theory that this bug existed prior to this commit, it just doesn't manifest with the same
seed prior to this.

Going one commit prior to when the randomization changed (80a6590c52b) I then attempted to
beast this test w/diff seeds to try and find a new permutation that would fail...
{noformat}
ant beast -Dbeast.iters=1000  -Dtestcase=SpellCheckCollatorTest -Dtests.method=testEstimatedHitCounts
-Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.badapples=true  -Dtests.locale=et-EE
-Dtests.timezone=Australia/Canberra -Dtests.asserts=true -Dtests.file.encoding=UTF-8
{noformat}
...and found this...
{noformat}
  [beaster]   2> 5212 INFO  (TEST-SpellCheckCollatorTest.testEstimatedHitCounts-seed#[989A055D5EA05EC3])
[    ] o.a.s.SolrTestCaseJ4 ###Ending testEstimatedHitCounts
  [beaster]   2> NOTE: reproduce with: ant test  -Dtestcase=SpellCheckCollatorTest -Dtests.method=testEstimatedHitCounts
-Dtests.seed=989A055D5EA05EC3 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true
-Dtests.badapples=true -Dtests.locale=et-EE -Dtests.timezone=Australia/Canberra -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
  [beaster] [14:21:03.465] ERROR   0.40s | SpellCheckCollatorTest.testEstimatedHitCounts <<<
  [beaster]    > Throwable #1: java.lang.RuntimeException: Exception during query
  [beaster]    > 	at __randomizedtesting.SeedInfo.seed([989A055D5EA05EC3:A921BB68FB9F4E13]:0)
  [beaster]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:934)
  [beaster]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:894)
  [beaster]    > 	at org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:569)
...
  [beaster]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/long[@name='hits'
and 3 <= . and . <= 13]
  [beaster]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
  [beaster]    > <response>
  [beaster]    > <lst name="responseHeader"><int name="status">0</int><int
name="QTime">3</int></lst><result name="response" numFound="0" start="0"></result><lst
name="spellcheck"><lst name="suggestions"><lst name="everother"><int name="numFound">1</int><int
name="startOffset">9</int><int name="endOffset">18</int><arr name="suggestion"><str>everyother</str></arr></lst></lst><lst
name="collations"><lst name="collation"><str name="collationQuery">teststop:everyother</str><long
name="hits">14</long><lst name="misspellingsAndCorrections"><str name="everother">everyother</str></lst></lst></lst></lst>
  [beaster]    > </response>
  [beaster]    > 
  [beaster]    > 	request was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=/spellCheckCompRH&q=teststop:everother&spellcheck.collateMaxCollectDocs=5
  [beaster]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:927)
  [beaster]    > 	... 41 more
{noformat}
A quick check confirmed that this seed reproduced even as far back as branch_8_0 (93d1e67886e75623f3f72526284bdbc3b1fef7e6),
but did *NOT* reproduce when this assert was last (substantively) modified (213a2a1791e4557afd2542a25e94eec65e29a42d)
as part of SOLR-5344 (in 2016)

So back to git bisect...
{noformat}
93d1e67886e75623f3f72526284bdbc3b1fef7e6 is the first bad commit
commit 93d1e67886e75623f3f72526284bdbc3b1fef7e6
Author: jimczi <jimczi@apache.org>
Date:   Mon Jan 28 22:43:20 2019 +0100

    LUCENE-8660: TopDocsCollectors now return an accurate count (instead of a lower bound)
if the total hit count is equal to the provided threshold.

:040000 040000 7bd5d8ab9e691e1253979cf8698fa88b2c7b9fb1 ec6791f9fc718f27eb7ffd9451bff3595e25250b
M	lucene
bisect run success
{noformat}
...I'm not really clear on how this change could affect ths test given that AFAIK every usage
of TopDocsCollectors in solr _should_ be insisting on an exact count of total hits (ie threshold=MAX_INT)
but since this commit has no modifications to the test-framework randomness that could otherwise
impact this test, the fact that it's first failing seed means *something* about these changes
impacted the underlying code being tested.

> SpellCheckCollatorTest.testEstimatedHitCounts reproducing failure seed
> ----------------------------------------------------------------------
>
>                 Key: SOLR-13946
>                 URL: https://issues.apache.org/jira/browse/SOLR-13946
>             Project: Solr
>          Issue Type: Bug
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Chris M. Hostetter
>            Priority: Major
>
> The following seed reliably fails on branch_8x.  based on the results of git bisect,
it appears that this in some way relates to changes made in LUCENE-8660 (TopDocsCollectors's
accuracy in reporting totalHits when dealing with totalHitsThreshold) even though all usage
in sol should be requesting {{totalHitsThreshold=Integer.MAX_VALUE}}
> {noformat}
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SpellCheckCollatorTest
-Dtests.method=testEstimatedHitCounts -Dtests.seed=AFA731DEE618DA14 -Dtests.multiplier=2 -Dtests.nightly=true
-Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ga-IE -Dtests.timezone=Canada/Atlantic
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
>    [junit4] ERROR   0.36s | SpellCheckCollatorTest.testEstimatedHitCounts <<<
>    [junit4]    > Throwable #1: java.lang.RuntimeException: Exception during query
>    [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AFA731DEE618DA14:9E1C8FEB4327CAC4]:0)
>    [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1001)
>    [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:961)
>    [junit4]    > 	at org.apache.solr.spelling.SpellCheckCollatorTest.testEstimatedHitCounts(SpellCheckCollatorTest.java:569)
>    [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
>    [junit4]    > Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=//lst[@name='spellcheck']/lst[@name='collations']/lst[@name='collation']/long[@name='hits'
and 3 <= . and . <= 13]
>    [junit4]    > 	xml response was: <?xml version="1.0" encoding="UTF-8"?>
>    [junit4]    > <response>
>    [junit4]    > <lst name="responseHeader"><int name="status">0</int><int
name="QTime">2</int></lst><result name="response" numFound="0" start="0"></result><lst
name="spellcheck"><lst name="suggestions"><lst name="everother"><int name="numFound">1</int><int
name="startOffset">9</int><int name="endOffset">18</int><arr name="suggestion"><str>everyother</str></arr></lst></lst><lst
name="collations"><lst name="collation"><str name="collationQuery">teststop:everyother</str><long
name="hits">14</long><lst name="misspellingsAndCorrections"><str name="everother">everyother</str></lst></lst></lst></lst>
>    [junit4]    > </response>
>    [junit4]    > 	request was:spellcheck=true&spellcheck.dictionary=direct&spellcheck.count=1&spellcheck.collate=true&spellcheck.maxCollationTries=1&spellcheck.maxCollations=1&spellcheck.collateExtendedResults=true&qt=/spellCheckCompRH&q=teststop:everother&spellcheck.collateMaxCollectDocs=5
>    [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:994)
>    [junit4]    > 	... 41 more
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Mime
View raw message