Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8174717EE1 for ; Thu, 17 Sep 2015 13:44:06 +0000 (UTC) Received: (qmail 19798 invoked by uid 500); 17 Sep 2015 13:44:03 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 19758 invoked by uid 500); 17 Sep 2015 13:44:03 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 19748 invoked by uid 99); 17 Sep 2015 13:44:03 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Sep 2015 13:44:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 25735C084E for ; Thu, 17 Sep 2015 13:44:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.88 X-Spam-Level: ** X-Spam-Status: No, score=2.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id elAMgB9jwfLt for ; Thu, 17 Sep 2015 13:43:58 +0000 (UTC) Received: from mail-vk0-f54.google.com (mail-vk0-f54.google.com [209.85.213.54]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 41FBA20864 for ; Thu, 17 Sep 2015 13:43:58 +0000 (UTC) Received: by vkfp126 with SMTP id p126so10874285vkf.3 for ; Thu, 17 Sep 2015 06:43:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=bqGpbB/RTbl9+Sq7TPfcBV/b3MR3rY4WwvFm8GLvcdk=; b=Qc0WP04TQCZ8vi6Ez/j4uVRTg6YXiRKwUFpBjQsObBCHIg9lyMsMM/y7usb64ubbZg V3cf6ubwOg7KnS3q7z7i/fvKhlNmxPYwWzEETUdSXzaqFjaB7jKJWDdtpgc3Y0ELjJan 15UBHC+7nJJVGxkVx39gSqJXW1HnVDE7e1M+uL8+GuL4WgGKS7OisWq9/haNNbpAhFQy mvG7t5RW1hl0R0aevHWIzoy4xTT6bgk8J+3nKuClRpW4O5Yjao6zBXKdQc6jquSg1PWV VNM685oVRONphEKxFUAUzsjyO2iff/HLOcOYrIK7trB2IC94WyiczqW7Id7xj0pd9i0g nY6A== MIME-Version: 1.0 X-Received: by 10.31.15.66 with SMTP id 63mr35727079vkp.147.1442497430789; Thu, 17 Sep 2015 06:43:50 -0700 (PDT) Received: by 10.31.94.200 with HTTP; Thu, 17 Sep 2015 06:43:50 -0700 (PDT) In-Reply-To: References: Date: Thu, 17 Sep 2015 14:43:50 +0100 Message-ID: Subject: Re: Upgrade Limitations Question From: Vasileios Vlachos To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11435db0f3a278051ff19aa5 --001a11435db0f3a278051ff19aa5 Content-Type: text/plain; charset=UTF-8 Thank you very much for pointing this out Victor. Really useful to know. On Wed, Sep 16, 2015 at 4:55 PM, Victor Chen wrote: > Yes, you can examine the actual sstables in your cassandra data dir. That > will tell you what version sstables you have on that node. > > You can refer to this link: > http://www.bajb.net/2013/03/cassandra-sstable-format-version-numbers/ > which I found via google search phrase "sstable versions" to see which > version you need to look for-- the relevant section of the link says: > >> Cassandra stores the version of the SSTable within the filename, >> following the format *Keyspace-ColumnFamily-(optional tmp >> marker-)SSTableFormat-generation* >> > > FYI-- and at least in the cassandra-2.1 branch of the source code-- you > can find sstable format generation descriptions in comments of > Descriptor.java. Looks like for your old and new versions, you'd be looking > for something like: > > for 1.2.1: > $ find -name "*-ib-*" -ls > > for 2.0.1: > $ find -name "*-jb-*" -ls > > > On Wed, Sep 16, 2015 at 10:02 AM, Vasileios Vlachos < > vasileiosvlachos@gmail.com> wrote: > >> >> Hello Rob and thanks for your reply, >> >> At the end we had to wait for the upgradesstables ti finish on every >> node. Just to eliminate the possibility of this being the reason of any >> weird behaviour after the upgrade. However, this process might take a long >> time in a cluster with a large number of nodes which means no new work can >> be done for that period. >> >> 1) TRUNCATE requires all known nodes to be available to succeed, if you >>> are restarting one, it won't be available. >>> >> >> I suppose all means all, not all replicas here, is that right? Not >> directly related to the original question, but that might explain why we >> end up with peculiar behaviour some times when we run TRUNCATE. We've now >> taken the approach DROP it and do it again when possible (even though this >> is still problematic when using the same CF name). >> >> >>> 2) in theory, the newly upgraded nodes might not get the DDL schema >>> update properly due to some incompatible change >>> >>> To check for 2, do : >>> " >>> nodetool gossipinfo | grep SCHEMA |sort | uniq -c | sort -n >>> " >>> >>> Before and after and make sure the schema propagates correctly. There >>> should be a new version on all nodes between each DDL change, if there is >>> you will likely be able to see the new schema on all the new nodes. >>> >>> >> Yeas, this makes perfect sense. We monitor the schema changes every >> minutes across the cluster with Nagios by checking the JMX console. It is >> an important thing to monitor in several situations (running migrations for >> example, or during upgrades like you describe here). >> >> Is there a way to find out if the upgradesstables has been run against a >> particular node or not? >> >> Many Thanks, >> Vasilis >> > > --001a11435db0f3a278051ff19aa5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thank you very much for pointing this out Victor. Really u= seful to know.

On Wed, Sep 16, 2015 at 4:55 PM, Victor Chen <victor.h.chen@gmail= .com> wrote:
Ye= s, you can examine the actual sstables in your cassandra data dir. That wil= l tell you what version sstables you have on that node.

You can refe= r to this link: http://www.bajb.net/2013/03/cassan= dra-sstable-format-version-numbers/ which I found via google search phr= ase "sstable versions" to see which version you need to look for-= - the relevant section of the link says:
Cassandra stores the version of the SSTable within the filename= , following the format=C2=A0Keyspace-Col= umnFamily-(optional tmp marker-)SSTableFormat-generation

FYI-- and at lea= st in the cassandra-2.1 branch of the source code-- you can find sstable fo= rmat generation descriptions in comments of Descriptor.java. Looks like for= your old and new versions, you'd be looking for something like:
for 1.2.1:
$ find <path to datadir> -name "*-ib-*" -ls<= br>
for 2.0.1:
$ find <path to datadir> -name "*-jb-*"= ; -ls


On Wed, Sep 16, 2015 at 10:= 02 AM, Vasileios Vlachos <vasileiosvlachos@gmail.com> wrote:

Hello Rob and thanks for your r= eply,

= At the end we had to wait for the upgradesstables ti finish on every node. = Just to eliminate the possibility of this being the reason of any weird beh= aviour after the upgrade. However, this process might take a long time in a= cluster with a large number of nodes which means no new work can be done f= or that period.=C2=A0

1) TRUNCATE requires all known nodes to be available to succeed, if you = are restarting one, it won't be available.

I suppose all means all, not all replica= s here, is that right? Not directly related to the original question, but t= hat might explain why we end up with peculiar behaviour some times when we = run TRUNCATE. We've now taken the approach DROP it and do it again when= possible (even though this is still problematic when using the same CF nam= e).
=C2=A0
2) in theor= y, the newly upgraded nodes might not get the DDL schema update properly du= e to some incompatible change

To check for 2, do :=
"
nodetool gossipinfo | grep SCHEMA |sort | uniq = -c | sort -n
"

Before and after and= make sure the schema propagates correctly. There should be a new version o= n all nodes between each DDL change, if there is you will likely be able to= see the new schema on all the new nodes.

<= /div>

Yeas, this makes perfect sense= . We monitor the schema changes every minutes across the cluster with Nagio= s by checking the JMX console. It is an important thing to monitor in sever= al situations (running migrations for example, or during upgrades like you = describe here).
=C2=A0
Is there a way to find out if th= e upgradesstables has been run against a particular node or not?=C2=A0

Many Thanks,
Vasilis


--001a11435db0f3a278051ff19aa5--