cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Russ Hatch (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-12171) counter mismatch during rolling upgrade from 2.2 to 3.0
Date Mon, 11 Jul 2016 19:59:11 GMT
Russ Hatch created CASSANDRA-12171:
--------------------------------------

             Summary: counter mismatch during rolling upgrade from 2.2 to 3.0
                 Key: CASSANDRA-12171
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12171
             Project: Cassandra
          Issue Type: Bug
            Reporter: Russ Hatch
            Assignee: Aleksey Yeschenko


This may occur on other versions, but 3.0 is where I observed it recently.

N=RF=3, counter writes at quorum, reads at quorum.

This is being seen on some upgrade tests I'm currently repairing here: https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix
(this branch is to resolve an issue where counters were not being properly tested during rolling
upgrade tests).

The test runs a continuous counter incrementing process, as well as a continuous counter checking
process. Once a counter value has been verified, the test code makes it eligible to be incremented
again.

The test is encountering the problem when trying to check an expected counter value and not
matching expectations, for example:
{noformat}
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/rhatch/git/cstar/cassandra-dtest/upgrade_tests/upgrade_through_versions_test.py",
line 210, in counter_checker
    tester.assertEqual(expected_count, actual_count)
  File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: 1 != 2
ERROR
{noformat}

To check if something else could be going on, I did an experiment where I changed the test
to not upgrade nodes (just drain, stop, start) and the mismatch didn't occur in several attempts.
So it appears something about upgrading is possibly the culprit.

To run the test and repro locally:
{noformat}
grab my dtest branch at https://github.com/riptano/cassandra-dtest/tree/upgrade_counters_fix

export UPGRADE_TEST_RUN=true

nosetests -v upgrade_tests/upgrade_through_versions_test.py:TestUpgrade_current_2_2_x_To_indev_3_0_x.rolling_upgrade_test
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message