Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0534C10B2D for ; Thu, 15 Jan 2015 16:42:34 +0000 (UTC) Received: (qmail 4179 invoked by uid 500); 15 Jan 2015 16:42:35 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 4144 invoked by uid 500); 15 Jan 2015 16:42:35 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 4120 invoked by uid 99); 15 Jan 2015 16:42:35 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Jan 2015 16:42:35 +0000 Date: Thu, 15 Jan 2015 16:42:35 +0000 (UTC) From: "Lennart Liberg (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (CASSANDRA-8628) Stalling rebuild of new 2.1.12 node from cluster running 2.0.9 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lennart Liberg updated CASSANDRA-8628: -------------------------------------- Description: We wanted to add a new datacenter to our cluster consisting of two datacenters containing 2 nodes and 1 node respectively, running version 2.0.9 We configured a node running cassandra 2.1.2 in the new datacenter (auto_bootstrap: false) and verified that it managed to join the cluster as expected (`nodetool status` saying its status was UN). Upon executing `nodetool rebuild ams1`, the system.log output is observed, with a new UnknownColumnFamilyException and NullPointerException every few minutes, and no data being transfered. After downgrading to cassandra 2.0.11, the rebuild operation worked flawless, using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml file (a lot of new items in 2.1.x). My reading of the upgrading compatibility document leads me to think that there should not be any problem in performing this kind of rebuild. http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html system.log output: {noformat} INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228 StorageService.java:958 - rebuild from dc: ams1 INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305 StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Executing streaming plan for Rebuild INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306 StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Starting streaming to 37.xx.xx.1/37.xx.xx.1 INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610 StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342, ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1 WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 - UnknownColumnFamilyException reading from socket; closing org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4 at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2] ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 - Exception in thread Thread[Thread-28974,5,main] java.lang.NullPointerException: null at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2] {noformat} was: We wanted to add a new datacenter to our cluster consisting of two datacenters containing 2 nodes and 1 node respectively, running version 2.0.9 We configured a node running cassandra 2.1.2 in the new datacenter (auto_bootstrap: false) and verified that it managed to join the cluster as expected (`nodetool status` saying its status was UN). Upon executing `nodetool rebuild ams1`, the system.log output is observed, with a new UnknownColumnFamilyException and NullPointerException every few minutes, and no data being transfered. After downgrading to cassandra 2.0.11, the rebuild operation worked flawless, using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml file (a lot of new items in 2.1.x). My reading of the upgrading compatibility document does not lead me to think there ought to be any problem in performing this kind of rebuild. http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html system.log output: {noformat} INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228 StorageService.java:958 - rebuild from dc: ams1 INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305 StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Executing streaming plan for Rebuild INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306 StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Starting streaming to 37.xx.xx.1/37.xx.xx.1 INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610 StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342, ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1 WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 - UnknownColumnFamilyException reading from socket; closing org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4 at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2] ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 - Exception in thread Thread[Thread-28974,5,main] java.lang.NullPointerException: null at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2] {noformat} > Stalling rebuild of new 2.1.12 node from cluster running 2.0.9 > -------------------------------------------------------------- > > Key: CASSANDRA-8628 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8628 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM > Cluster running cassandra 2.0.9, new node running 2.1.2, > inter-node encryption: all > Reporter: Lennart Liberg > > We wanted to add a new datacenter to our cluster consisting of two datacenters containing 2 nodes and 1 node respectively, running version 2.0.9 > We configured a node running cassandra 2.1.2 in the new datacenter (auto_bootstrap: false) and verified that it managed to join the cluster as expected (`nodetool status` saying its status was UN). > Upon executing `nodetool rebuild ams1`, the system.log output is observed, with a new UnknownColumnFamilyException and NullPointerException every few minutes, and no data being transfered. > After downgrading to cassandra 2.0.11, the rebuild operation worked flawless, using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml file (a lot of new items in 2.1.x). > My reading of the upgrading compatibility document leads me to think that there should not be any problem in performing this kind of rebuild. > http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html > system.log output: > {noformat} > INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228 StorageService.java:958 - rebuild from dc: ams1 > INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305 StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Executing streaming plan for Rebuild > INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306 StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Starting streaming to 37.xx.xx.1/37.xx.xx.1 > INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610 StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342, ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1 > WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 - UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4 > at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2] > ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 - Exception in thread Thread[Thread-28974,5,main] > java.lang.NullPointerException: null > at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2] > at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2] > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)