incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Heffner <m...@librato.com>
Subject Re: Upgrade 1.1.2 -> 1.1.6
Date Tue, 20 Nov 2012 15:47:48 GMT
Alain,

My understanding is that drain ensures that all memtables are flushed, so
that there is no data in the commitlog that is isn't in an sstable. A
marker is saved that indicates the commit logs should not be replayed.
Commitlogs are only removed from disk periodically
(after commitlog_total_space_in_mb is exceeded?).

With 1.1.5/6, all nanotime commitlogs are replayed on startup regardless of
whether they've been flushed. So in our case manually removing all the
commitlogs after a drain was the only way to prevent their replay.

Mike




On Tue, Nov 20, 2012 at 5:19 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

> @Mike
>
> I am glad to see I am not the only one with this issue (even if I am sorry
> it happened to you of course.).
>
> Isn't drain supposed to clear the commit logs ? Did removing them worked
> properly ?
>
> I his warning to C* users, Jonathan Ellis told that a drain would avoid
> this issue, It seems like it doesn't.
>
> @Rob
>
> You understood precisely the 2 issues I met during the upgrade. I am sad
> to see none of them is yet resolved and probably wont.
>
>
> 2012/11/20 Mike Heffner <mike@librato.com>
>
>> Alain,
>>
>> We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs
>> replayed regardless of the drain. After noticing this on the first node, we
>> did the following:
>>
>> * nodetool flush
>> * nodetool drain
>> * service cassandra stop
>> * mv /path/to/logs/*.log /backup/
>> * apt-get install cassandra
>> <restarts automatically>
>>
>> I also agree that starting C* after an upgrade/install seems quite broken
>> if it was already stopped before the install. However annoying, I have
>> found this to be the default for most Ubuntu daemon packages.
>>
>> Mike
>>
>>
>> On Thu, Nov 15, 2012 at 9:21 AM, Alain RODRIGUEZ <arodrime@gmail.com>wrote:
>>
>>> We had an issue with counters over-counting even using the nodetool
>>> drain command before upgrading...
>>>
>>> Here is my bash history
>>>
>>>    69  cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>>    70  cp /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>>    71  sudo apt-get install cassandra
>>>    72  nodetool disablethrift
>>>    73  nodetool drain
>>>    74  service cassandra stop
>>>    75  cat /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>>    76  vim /etc/cassandra/cassandra-env.sh
>>>    77  cat /etc/cassandra/cassandra.yaml
>>> /etc/cassandra/cassandra.yaml.bak
>>>    78  vim /etc/cassandra/cassandra.yaml
>>>    79  service cassandra start
>>>
>>> So I think I followed these steps
>>> http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>>>
>>> I merged my conf files with an external tool so consider I merged my
>>> conf files on steps 76 and 78.
>>>
>>> I saw that the "sudo apt-get install cassandra" stop the server and
>>> restart it automatically. So it updated without draining and restart before
>>> I had the time to reconfigure the conf files. Is this "normal" ? Is there a
>>> way to avoid it ?
>>>
>>> So for the second node I decided to try to stop C*before the upgrade.
>>>
>>>   125  cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>>   126  cp /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>>   127  nodetool disablegossip
>>>   128  nodetool disablethrift
>>>   129  nodetool drain
>>>   130  service cassandra stop
>>>   131  sudo apt-get install cassandra
>>>
>>> //131 : This restarted cassandra
>>>
>>>   132  nodetool disablethrift
>>>   133  nodetool disablegossip
>>>   134  nodetool drain
>>>   135  service cassandra stop
>>>   136  cat /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>>   137  cim /etc/cassandra/cassandra-env.sh
>>>   138  vim /etc/cassandra/cassandra-env.sh
>>>   139  cat /etc/cassandra/cassandra.yaml
>>> /etc/cassandra/cassandra.yaml.bak
>>>   140  vim /etc/cassandra/cassandra.yaml
>>>   141  service cassandra start
>>>
>>> After both of these updates I saw my current counters increase without
>>> any reason.
>>>
>>> Did I do anything wrong ?
>>>
>>> Alain
>>>
>>>
>>
>>
>> --
>>
>>   Mike Heffner <mike@librato.com>
>>   Librato, Inc.
>>
>>
>>
>


-- 

  Mike Heffner <mike@librato.com>
  Librato, Inc.

Mime
View raw message