incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Upgrade 1.1.2 -> 1.1.6
Date Sun, 18 Nov 2012 20:15:58 GMT
> time (UTC)	       0	1	2	3	4	5	6	7	8	9	10	11	12	13
> Good value       88	44	26	35	26	86	187	251	455	389	473	367	453	373
> C* counter        149	82	45	68	38	146	329	414	746	566	473	377	453	373
> I finished my Cassandra 1.1.6 upgrades at 9:30 UTC.


This looks like the counters were more out of sync before the upgrade than after?
Do you know if your client is retrying counter operations ? (I saw some dropped messages in
the S1 log). 

S1 shows a lot of Commit Log replay going on. Reading your timeline below this sounds like
the auto restart catching you out. 

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 16/11/2012, at 10:22 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

> Here is an example of the increase for some counter (counting events per hour)
> 
> time (UTC)	       0	1	2	3	4	5	6	7	8	9	10	11	12	13
> Good value       88	44	26	35	26	86	187	251	455	389	473	367	453	373
> C* counter        149	82	45	68	38	146	329	414	746	566	473	377	453	373
> 
> I finished my Cassandra 1.1.6 upgrades at 9:30 UTC.
> 
> I found wrong values since the day before at 20:00 UTC (counters from hours before are
good)
> 
> Here are the logs from the output:
> Server 1: http://pastebin.com/WyCm6Ef5 (This one is from the same server as the first
bash history on my first mail)
> Server 2: http://pastebin.com/gBe2KL2b  (This one is from the same server as the second
bash history on my first mail)
> 
> Alain
> 
> 2012/11/15 aaron morton <aaron@thelastpickle.com>
> Can you provide an example of the increase ? 
> 
> Can you provide the log from startup ?
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 16/11/2012, at 3:21 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
> 
>> We had an issue with counters over-counting even using the nodetool drain command
before upgrading...
>> 
>> Here is my bash history
>> 
>>    69  cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>    70  cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
>>    71  sudo apt-get install cassandra
>>    72  nodetool disablethrift
>>    73  nodetool drain
>>    74  service cassandra stop
>>    75  cat /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
>>    76  vim /etc/cassandra/cassandra-env.sh
>>    77  cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>    78  vim /etc/cassandra/cassandra.yaml
>>    79  service cassandra start
>> 
>> So I think I followed these steps http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps
>> 
>> I merged my conf files with an external tool so consider I merged my conf files on
steps 76 and 78.
>> 
>> I saw that the "sudo apt-get install cassandra" stop the server and restart it automatically.
So it updated without draining and restart before I had the time to reconfigure the conf files.
Is this "normal" ? Is there a way to avoid it ?
>> 
>> So for the second node I decided to try to stop C*before the upgrade.
>> 
>>   125  cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>   126  cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
>>   127  nodetool disablegossip
>>   128  nodetool disablethrift
>>   129  nodetool drain
>>   130  service cassandra stop
>>   131  sudo apt-get install cassandra
>> 
>> //131 : This restarted cassandra
>> 
>>   132  nodetool disablethrift
>>   133  nodetool disablegossip
>>   134  nodetool drain
>>   135  service cassandra stop
>>   136  cat /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak
>>   137  cim /etc/cassandra/cassandra-env.sh
>>   138  vim /etc/cassandra/cassandra-env.sh
>>   139  cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
>>   140  vim /etc/cassandra/cassandra.yaml
>>   141  service cassandra start
>> 
>> After both of these updates I saw my current counters increase without any reason.
>> 
>> Did I do anything wrong ?
>> 
>> Alain
>> 
> 
> 


Mime
View raw message