lucene-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris M. Hostetter (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (SOLR-13864) MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed
Date Tue, 12 Nov 2019 21:07:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972758#comment-16972758
] 

Chris M. Hostetter edited comment on SOLR-13864 at 11/12/19 9:06 PM:
---------------------------------------------------------------------

-Commit 603be023feaf3f8e3e739e532b488068710d9097 in lucene-solr's branch refs/heads/master
from Chris M. Hostetter-
 -[ [https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=603be02] ]-

-SOLR-13864: SolrTestCaseJ4.getNextAvailablePort() has been deprecated-

Mistaken Jira tagging in commit message


was (Author: jira-bot):
Commit 603be023feaf3f8e3e739e532b488068710d9097 in lucene-solr's branch refs/heads/master
from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=603be02 ]

SOLR-13864: SolrTestCaseJ4.getNextAvailablePort() has been deprecated


> MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization
beyond test seed
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: SOLR-13864
>                 URL: https://issues.apache.org/jira/browse/SOLR-13864
>             Project: Solr
>          Issue Type: Test
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Chris M. Hostetter
>            Assignee: Joel Bernstein
>            Priority: Major
>         Attachments: apache_Lucene-Solr-BadApples-Tests-master_531.log.txt
>
>
> We're seeing a a fairly steady trickle of MathExpressionTest from various jenkins boxes
going back quite a while ... mostly from testGammaDistribution, but other tests pop up now
and then.
> the crux of the problem with this test seems to break down into 2 categories:
>  # tests that make assumptions about the relative values that will come out of taking
samples from different random distributions that aren't garunteed to be true
>  ** ie: comparing 2 random samples from 2 diff shaped gamma distributions and expecting
one to always be strictly greater then the other. I'm not a stats guy, but my naive understanding
is that on the low end some of these shapes may cross over, so every possible random sample
from one shape is not garunteed to be less then every ossible random sample from a diff shape
>  # the code being tested does it's own randomization outside of the crontrol of the test
framework (or test client)
>  ** this causes the seeds to not reproduce
> ----
> Tests should not be making assertions about random data that aren't 100% garunteed to
be true in all cases (ie: {{random().nextInt(5) < (5.0D + (double) random().nextInt(5))}}
is one thing, {{random().nextInt(5) < (4.99999D + (double) random().nextInt(5))}} is a
diff story.
> Randomized behavior in solr (non-test) code should ideally have some way for being controlled
by the client/tests ... either via a request param used to initialize any new Random instances,
or for example the use of the "tests.seed" property in various places in the code to try and
provide some reproducibility even when the external solr client isn't even aware of randomization
being a factor in the behavior of the code.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Mime
View raw message