Alain,
My understanding is that drain ensures that all memtables are flushed, so
that there is no data in the commitlog that is isn't in an sstable. A
marker is saved that indicates the commit logs should not be replayed.
Commitlogs are only removed from disk periodically
(after commitlog_total_space_in_mb is exceeded?).
With 1.1.5/6, all nanotime commitlogs are replayed on startup regardless of
whether they've been flushed. So in our case manually removing all the
commitlogs after a drain was the only way to prevent their replay.
Mike
On Tue, Nov 20, 2012 at 5:19 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
> @Mike
>
> I am glad to see I am not the only one with this issue (even if I am sorry
> it happened to you of course.).
>
> Isn't drain supposed to clear the commit logs ? Did removing them worked
> properly ?
>
> I his warning to C* users, Jonathan Ellis told that a drain would avoid
> this issue, It seems like it doesn't.
>
> @Rob
>
> You understood precisely the 2 issues I met during the upgrade. I am sad
> to see none of them is yet resolved and probably wont.
>
>
> 2012/11/20 Mike Heffner <mike@librato.com>
>
>> Alain,
>>
>> We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs
>> replayed regardless of the drain. After noticing this on the first node, we
>> did the following:
>>
>> * nodetool flush
>> * nodetool drain
>> * service cassandra stop
>> * mv /path/to/logs/*.log /backup/
>> * apt-get install cassandra
>> <restarts automatically>
>>
>> I also agree that starting C* after an upgrade/install seems quite broken
>> if it was already stopped before the install. However annoying, I have
>> found this to be the default for most Ubuntu daemon packages.
>>
>> Mike
>>
>>
>> On Thu, Nov 15, 2012 at 9:21 AM, Alain RODRIGUEZ <arodrime@gmail.com>wrote:
>>
>>> We had an issue with counters over-counting even using the nodetool
>>> drain command before upgrading...
>>>
>>> Here is my bash history
>>>
>>> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>> 70 cp /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>> 71 sudo apt-get install cassandra
>>> 72 nodetool disablethrift
>>> 73 nodetool drain
>>> 74 service cassandra stop
>>> 75 cat /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>> 76 vim /etc/cassandra/cassandra-env.sh
>>> 77 cat /etc/cassandra/cassandra.yaml
>>> /etc/cassandra/cassandra.yaml.bak
>>> 78 vim /etc/cassandra/cassandra.yaml
>>> 79 service cassandra start
>>>
>>> So I think I followed these steps
>>> http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>>>
>>> I merged my conf files with an external tool so consider I merged my
>>> conf files on steps 76 and 78.
>>>
>>> I saw that the "sudo apt-get install cassandra" stop the server and
>>> restart it automatically. So it updated without draining and restart before
>>> I had the time to reconfigure the conf files. Is this "normal" ? Is there a
>>> way to avoid it ?
>>>
>>> So for the second node I decided to try to stop C*before the upgrade.
>>>
>>> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>> 126 cp /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>> 127 nodetool disablegossip
>>> 128 nodetool disablethrift
>>> 129 nodetool drain
>>> 130 service cassandra stop
>>> 131 sudo apt-get install cassandra
>>>
>>> //131 : This restarted cassandra
>>>
>>> 132 nodetool disablethrift
>>> 133 nodetool disablegossip
>>> 134 nodetool drain
>>> 135 service cassandra stop
>>> 136 cat /etc/cassandra/cassandra-env.sh
>>> /etc/cassandra/cassandra-env.sh.bak
>>> 137 cim /etc/cassandra/cassandra-env.sh
>>> 138 vim /etc/cassandra/cassandra-env.sh
>>> 139 cat /etc/cassandra/cassandra.yaml
>>> /etc/cassandra/cassandra.yaml.bak
>>> 140 vim /etc/cassandra/cassandra.yaml
>>> 141 service cassandra start
>>>
>>> After both of these updates I saw my current counters increase without
>>> any reason.
>>>
>>> Did I do anything wrong ?
>>>
>>> Alain
>>>
>>>
>>
>>
>> --
>>
>> Mike Heffner <mike@librato.com>
>> Librato, Inc.
>>
>>
>>
>
--
Mike Heffner <mike@librato.com>
Librato, Inc.
|