@Mike

I am glad to see I am not the only one with this issue (even if I am sorry it happened to you of course.).

Isn't drain supposed to clear the commit logs ? Did removing them worked properly ?

I his warning to C* users, Jonathan Ellis told that a drain would avoid this issue, It seems like it doesn't.

@Rob

You understood precisely the 2 issues I met during the upgrade. I am sad to see none of them is yet resolved and probably wont.


2012/11/20 Mike Heffner <mike@librato.com>
Alain,

We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs replayed regardless of the drain. After noticing this on the first node, we did the following:

* nodetool flush
* nodetool drain
* service cassandra stop
* mv /path/to/logs/*.log /backup/
* apt-get install cassandra
<restarts automatically>

I also agree that starting C* after an upgrade/install seems quite broken if it was already stopped before the install. However annoying, I have found this to be the default for most Ubuntu daemon packages.

Mike


On Thu, Nov 15, 2012 at 9:21 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
We had an issue with counters over-counting even using the nodetool drain command before upgrading...

Here is my bash history

   69  cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
   70  cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
   71  sudo apt-get install cassandra
   72  nodetool disablethrift
   73  nodetool drain
   74  service cassandra stop
   75  cat /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
   76  vim /etc/cassandra/cassandra-env.sh
   77  cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
   78  vim /etc/cassandra/cassandra.yaml
   79  service cassandra start


I merged my conf files with an external tool so consider I merged my conf files on steps 76 and 78.

I saw that the "sudo apt-get install cassandra" stop the server and restart it automatically. So it updated without draining and restart before I had the time to reconfigure the conf files. Is this "normal" ? Is there a way to avoid it ?

So for the second node I decided to try to stop C*before the upgrade.

  125  cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
  126  cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
  127  nodetool disablegossip
  128  nodetool disablethrift
  129  nodetool drain
  130  service cassandra stop
  131  sudo apt-get install cassandra

//131 : This restarted cassandra

  132  nodetool disablethrift
  133  nodetool disablegossip
  134  nodetool drain
  135  service cassandra stop
  136  cat /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
  137  cim /etc/cassandra/cassandra-env.sh
  138  vim /etc/cassandra/cassandra-env.sh
  139  cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
  140  vim /etc/cassandra/cassandra.yaml
  141  service cassandra start

After both of these updates I saw my current counters increase without any reason.

Did I do anything wrong ?

Alain




--

  Mike Heffner <mike@librato.com>
  Librato, Inc.