cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9161) Add random interleaving for flush/compaction when running CQL unit tests
Date Thu, 03 Mar 2016 10:43:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15177651#comment-15177651
] 

Sylvain Lebresne commented on CASSANDRA-9161:
---------------------------------------------

bq. the problems we faced with upgrade tests due to the randomness of how the data is distributed

Yeah, but that's a bad example: that randomness was not properly controlled, it was making
things hard to reproduce and that was the problem. But this is not what I want to do here,
this will _not_ make tests non reproducible at all.

In testing, sometimes (always really, but that's a different subject) the state space is just
too big to be systematically explored on every run. Because of that, you do your best at exploring
the most meaningful part of the space (and I'm not saying we can't improve on that part btw,
we can and we should), but there is still space you can't explore.  Hoping you'll be so good
at finding the meaningful subset of the states to test so that no bug will lurk in the remaining
space is just wishful thinking. So this is just about getting incrementally better coverage
of the full space by using some new random state on every run _for the parts we can't reasonably
explore systematically_ (and I'm happy to discuss which parts can reasonably be explored systematically
and which aren't btw).

bq. For flushing, the main problem that I have seen is that only the read or write path for
memtables was tested not the one for SSTables

It's really more complex. Unless your test has a single insert, there isn't _just_ one path
for memtables and one for sstables. There is the cases where some data is in memtable and
some in sstables, where there is more than one sstable involved (and we can flush for every
insert, or only in some places), whether we compact before reading etc... I have seen bugs
in pretty much all of those cases (I'm genuinely not kidding: there has been cases with range
tombstone in particular where things got only messes up when data was flushed at specific
point and compaction was run before reading).

And here again, don't get me wrong: for some tests, there may be a clear place where we want
to systematically test both with and without flush because we want that to be tested every
time and that's fine, we can do it. But we just can't systematically test all combinations.

> Add random interleaving for flush/compaction when running CQL unit tests
> ------------------------------------------------------------------------
>
>                 Key: CASSANDRA-9161
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9161
>             Project: Cassandra
>          Issue Type: Test
>            Reporter: Sylvain Lebresne
>              Labels: retrospective_generated
>
> Most CQL tests don't bother flushing, which means that they overwhelmingly test the memtable
path and not the sstables one. A simple way to improve on that would be to make {{CQLTester}}
issue flushes and compactions randomly between statements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message