cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brandon Williams (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-8628) Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
Date Thu, 15 Jan 2015 16:55:36 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Brandon Williams resolved CASSANDRA-8628.
-----------------------------------------
    Resolution: Not a Problem

> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -------------------------------------------------------------
>
>                 Key: CASSANDRA-8628
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
>            Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two datacenters
containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter (auto_bootstrap: false)
and verified that it managed to join the cluster as expected (`nodetool status` saying its
status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is observed, with
a new UnknownColumnFamilyException and NullPointerException every few minutes, and no data
being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked flawlessly, using
a cassandra.yaml configured as identical as possible to the 2.1.2 yaml file (a lot of new
items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that there should
not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO  [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228 StorageService.java:958
- rebuild from dc: ams1
> INFO  [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305 StreamResultFuture.java:86
- [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Executing streaming plan for Rebuild
> INFO  [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306 StreamSession.java:213
- [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO  [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610 StreamCoordinator.java:209
- [Stream #91906440-9ccf-11e4-9cab-67625edb9342, ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN  [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 - UnknownColumnFamilyException
reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 - Exception in
thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
>         at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29011] 2015-01-15 16:07:58,349 CassandraDaemon.java:153 - Exception in
thread Thread[Thread-29011,5,main]
> java.lang.NullPointerException: null
>         at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29013] 2015-01-15 16:08:01,037 CassandraDaemon.java:153 - Exception in
thread Thread[Thread-29013,5,main]
> java.lang.NullPointerException: null
>         at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> WARN  [Thread-29065] 2015-01-15 16:10:56,374 IncomingTcpConnection.java:91 - UnknownColumnFamilyException
reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
~[apache-cassandra-2.1.2.jar:2.1.2]
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
~[apache-cassandra-2.1.2.jar:2.1.2]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message