cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
Date Thu, 15 Jan 2015 16:55:35 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278936#comment-14278936
] 

Brandon Williams commented on CASSANDRA-8628:
---------------------------------------------

bq. My reading of the upgrading compatibility document leads me to think that there should
not be any problem in performing this kind of rebuild.

Actually, there is.  Since the new node is 2.1, it won't pull schema from 2.0, which leads
to your "Couldn't find cfId" problems.  You should start the new node on 2.0, let it get the
schema, then upgrade it to 2.1.

> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -------------------------------------------------------------
>
>                 Key: CASSANDRA-8628
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>            Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter (auto_bootstrap: false)
and verified that it managed to join the cluster as expected (`nodetool status` saying its
status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is observed, with
a new UnknownColumnFamilyException and NullPointerException every few minutes, and no data
being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked flawlessly, using
a cassandra.yaml configured as identical as possible to the 2.1.2 yaml file (a lot of new
items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that there should
not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO  [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228 StorageService.java:958
- rebuild from dc: ams1
> INFO  [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305 StreamResultFuture.java:86
- [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Executing streaming plan for Rebuild
> INFO  [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306 StreamSession.java:213
- [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO  [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610 StreamCoordinator.java:209
- [Stream #91906440-9ccf-11e4-9cab-67625edb9342, ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN  [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 - UnknownColumnFamilyException
reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 - Exception in
thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
>         at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29011] 2015-01-15 16:07:58,349 CassandraDaemon.java:153 - Exception in
thread Thread[Thread-29011,5,main]
> java.lang.NullPointerException: null
>         at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29013] 2015-01-15 16:08:01,037 CassandraDaemon.java:153 - Exception in
thread Thread[Thread-29013,5,main]
> java.lang.NullPointerException: null
>         at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> WARN  [Thread-29065] 2015-01-15 16:10:56,374 IncomingTcpConnection.java:91 - UnknownColumnFamilyException
reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message