Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 316E7D0FA for ; Sun, 18 Nov 2012 20:16:26 +0000 (UTC) Received: (qmail 53319 invoked by uid 500); 18 Nov 2012 20:16:23 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 53292 invoked by uid 500); 18 Nov 2012 20:16:23 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 53284 invoked by uid 99); 18 Nov 2012 20:16:23 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Nov 2012 20:16:23 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a93.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Nov 2012 20:16:19 +0000 Received: from homiemail-a93.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a93.g.dreamhost.com (Postfix) with ESMTP id 9B50C8405B for ; Sun, 18 Nov 2012 12:15:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=PWGZE7GvWHoJCKpEGpDJvbbOK5 g=; b=WVtmnh+N56uBMRHiUVSON+w4Rv9q1ibep582elYYTc+8nJZvl+asFRAhXi LeNaUtqQ45vedEv4Y78deEj4+XFQoQ0g+X+v3hAw87UcOOquVsCtbSN8SAtEWCID SdjSSZRUXR5iDhiKsgd0UxIdR8rbQYlZqFyTeJVnmt1ZVvqGM= Received: from [192.168.2.13] (unknown [116.90.132.105]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a93.g.dreamhost.com (Postfix) with ESMTPSA id F402784059 for ; Sun, 18 Nov 2012 12:15:57 -0800 (PST) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_454D7875-F1A7-4D78-923C-AB455C1859B7" Message-Id: <41F43370-8B1D-4A42-8673-C540E47FE7BB@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Upgrade 1.1.2 -> 1.1.6 Date: Mon, 19 Nov 2012 09:15:58 +1300 References: <78EE60A6-3864-4F93-B7F8-BB481A121D41@thelastpickle.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_454D7875-F1A7-4D78-923C-AB455C1859B7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > time (UTC) 0 1 2 3 4 5 = 6 7 8 9 10 11 12 13 > Good value 88 44 26 35 26 86 187 = 251 455 389 473 367 453 373 > C* counter 149 82 45 68 38 146 329 = 414 746 566 473 377 453 373 > I finished my Cassandra 1.1.6 upgrades at 9:30 UTC. This looks like the counters were more out of sync before the upgrade = than after? Do you know if your client is retrying counter operations ? (I saw some = dropped messages in the S1 log).=20 S1 shows a lot of Commit Log replay going on. Reading your timeline = below this sounds like the auto restart catching you out.=20 Cheers =20 ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 16/11/2012, at 10:22 AM, Alain RODRIGUEZ wrote: > Here is an example of the increase for some counter (counting events = per hour) >=20 > time (UTC) 0 1 2 3 4 5 = 6 7 8 9 10 11 12 13 > Good value 88 44 26 35 26 86 187 = 251 455 389 473 367 453 373 > C* counter 149 82 45 68 38 146 329 = 414 746 566 473 377 453 373 >=20 > I finished my Cassandra 1.1.6 upgrades at 9:30 UTC. >=20 > I found wrong values since the day before at 20:00 UTC (counters from = hours before are good) >=20 > Here are the logs from the output: > Server 1: http://pastebin.com/WyCm6Ef5 (This one is from the same = server as the first bash history on my first mail) > Server 2: http://pastebin.com/gBe2KL2b (This one is from the same = server as the second bash history on my first mail) >=20 > Alain >=20 > 2012/11/15 aaron morton > Can you provide an example of the increase ?=20 >=20 > Can you provide the log from startup ? >=20 > Cheers >=20 > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand >=20 > @aaronmorton > http://www.thelastpickle.com >=20 > On 16/11/2012, at 3:21 AM, Alain RODRIGUEZ wrote: >=20 >> We had an issue with counters over-counting even using the nodetool = drain command before upgrading... >>=20 >> Here is my bash history >>=20 >> 69 cp /etc/cassandra/cassandra.yaml = /etc/cassandra/cassandra.yaml.bak >> 70 cp /etc/cassandra/cassandra-env.sh = /etc/cassandra/cassandra-env.sh.bak >> 71 sudo apt-get install cassandra >> 72 nodetool disablethrift >> 73 nodetool drain >> 74 service cassandra stop >> 75 cat /etc/cassandra/cassandra-env.sh = /etc/cassandra/cassandra-env.sh.bak >> 76 vim /etc/cassandra/cassandra-env.sh >> 77 cat /etc/cassandra/cassandra.yaml = /etc/cassandra/cassandra.yaml.bak >> 78 vim /etc/cassandra/cassandra.yaml >> 79 service cassandra start >>=20 >> So I think I followed these steps = http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps >>=20 >> I merged my conf files with an external tool so consider I merged my = conf files on steps 76 and 78. >>=20 >> I saw that the "sudo apt-get install cassandra" stop the server and = restart it automatically. So it updated without draining and restart = before I had the time to reconfigure the conf files. Is this "normal" ? = Is there a way to avoid it ? >>=20 >> So for the second node I decided to try to stop C*before the upgrade. >>=20 >> 125 cp /etc/cassandra/cassandra.yaml = /etc/cassandra/cassandra.yaml.bak >> 126 cp /etc/cassandra/cassandra-env.sh = /etc/cassandra/cassandra-env.sh.bak >> 127 nodetool disablegossip >> 128 nodetool disablethrift >> 129 nodetool drain >> 130 service cassandra stop >> 131 sudo apt-get install cassandra >>=20 >> //131 : This restarted cassandra >>=20 >> 132 nodetool disablethrift >> 133 nodetool disablegossip >> 134 nodetool drain >> 135 service cassandra stop >> 136 cat /etc/cassandra/cassandra-env.sh = /etc/cassandra/cassandra-env.sh.bak >> 137 cim /etc/cassandra/cassandra-env.sh >> 138 vim /etc/cassandra/cassandra-env.sh >> 139 cat /etc/cassandra/cassandra.yaml = /etc/cassandra/cassandra.yaml.bak >> 140 vim /etc/cassandra/cassandra.yaml >> 141 service cassandra start >>=20 >> After both of these updates I saw my current counters increase = without any reason. >>=20 >> Did I do anything wrong ? >>=20 >> Alain >>=20 >=20 >=20 --Apple-Mail=_454D7875-F1A7-4D78-923C-AB455C1859B7 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
time (UTC) 0 1 2 3 4 5 6 7 8 9 10 11 12 13
Good value   =     88 = 44 26 35 26 86 187 251 455 389 473 367 453 373
C* = counter   = 149 = 82 45 68 38 146 329 414 746 566 473 377 453 = 373
I finished = my Cassandra 1.1.6 upgrades at 9:30 = UTC.

This looks like the counters were = more out of sync before the upgrade than after?
Do you know if = your client is retrying counter operations ? (I saw some dropped = messages in the S1 log). 

S1 shows a lot = of Commit Log replay going on. Reading your timeline below this sounds = like the auto restart catching you = out. 

Cheers
 
http://www.thelastpickle.com

On 16/11/2012, at 10:22 AM, Alain RODRIGUEZ <arodrime@gmail.com> = wrote:

Here is an example of the increase for some counter = (counting events per hour)

time = (UTC) = 0 1 2 3 4 5 6 7 8 9 10 11 12 13
Good value       88 44 26 35 26 86 187 251 455 389 473 367 453 373
C* counter   = 149 82 45 68 38 146 329 414 746 566 473 377 453 373

I finished my Cassandra 1.1.6 upgrades at 9:30 = UTC.

I found wrong values since the day before = at 20:00 UTC (counters from hours before are = good)

Here are the logs from the output:
Server 1: http://pastebin.com/WyCm6Ef5 = (This one is from the same server as the first bash history on my first = mail)
Server 2: http://pastebin.com/gBe2KL2b&nbs= p; (This one is from the same server as the second bash history on my = first mail)

Alain

2012/11/15 = aaron morton <aaron@thelastpickle.com>
Can you provide an example of the = increase ? 

Can you provide the log from startup = ?

Cheers

-----------------
Aaron Morton
Freelance = Cassandra Developer
New = Zealand

@aaronmorton

On 16/11/2012, at 3:21 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

We had an issue with counters over-counting even using the = nodetool drain command before upgrading...

Here is my bash = history

   69  cp = /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
   70  cp /etc/cassandra/cassandra-env.sh = /etc/cassandra/cassandra-env.sh.bak
   71  sudo = apt-get install cassandra
   72  nodetool = disablethrift
   73  nodetool = drain
   74  service cassandra stop
   75  cat /etc/cassandra/cassandra-env.sh = /etc/cassandra/cassandra-env.sh.bak
   76  vim = /etc/cassandra/cassandra-env.sh
   77  cat = /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak
   78  vim = /etc/cassandra/cassandra.yaml
   79  service = cassandra start


I merged my conf files with an external tool = so consider I merged my conf files on steps 76 and = 78.

I saw that the "sudo apt-get install = cassandra" stop the server and restart it automatically. So it updated = without draining and restart before I had the time to reconfigure the = conf files. Is this "normal" ? Is there a way to avoid it ?

So for the second node I decided to try to stop = C*before the upgrade.

  125  cp = /etc/cassandra/cassandra.yaml = /etc/cassandra/cassandra.yaml.bak
  126  cp = /etc/cassandra/cassandra-env.sh = /etc/cassandra/cassandra-env.sh.bak
  127  nodetool disablegossip
  128 =  nodetool disablethrift
  129  nodetool = drain
  130  service cassandra stop
  = 131  sudo apt-get install cassandra

//131 : This restarted cassandra

  132  nodetool = disablethrift
  133  nodetool = disablegossip
  134  nodetool drain
  = 135  service cassandra stop
  136  cat = /etc/cassandra/cassandra-env.sh = /etc/cassandra/cassandra-env.sh.bak
  137  cim = /etc/cassandra/cassandra-env.sh
  138  vim = /etc/cassandra/cassandra-env.sh
  139  cat = /etc/cassandra/cassandra.yaml = /etc/cassandra/cassandra.yaml.bak
  140  vim = /etc/cassandra/cassandra.yaml
  141  service cassandra = start

After both of these updates I saw = my current counters increase without any = reason.

Did I do anything wrong = ?

Alain

=



= --Apple-Mail=_454D7875-F1A7-4D78-923C-AB455C1859B7--