cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10554) Auto Bootstraping a new node fails
Date Tue, 20 Oct 2015 14:24:27 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sylvain Lebresne updated CASSANDRA-10554:
-----------------------------------------
    Attachment: 0001-Add-debug.txt

I'm not sure this has anything to do with the column type. That error would typically happen
if the node receiving is not up to date on the schema for the table streamed, but a bootstrapping
node should be waiting on being up to date on schema before doing any streaming in theory
(or has that changed?).

In any case, if you can reproduce, can you apply the simple patch I attach (that only provide
some additional logging) on the boostrapped node, reproduce, and attach the full log for that
bootstrapped node? More details on what the test does would help too (if you could make it
a reproducible dtest that would be the best thing since sliced bread).

> Auto Bootstraping a new node fails
> ----------------------------------
>
>                 Key: CASSANDRA-10554
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10554
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Alan Boudreault
>             Fix For: 3.0.0
>
>         Attachments: 0001-Add-debug.txt
>
>
> I've been trying to add a new node in my 3.0 cluster and it seems to fail. All my nodes
are using apache/cassandra-3.0.0 branch. At the beginning, I can see the following error:
> {code}
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare completed.
Receiving 42 files(1910066622 bytes), sending 0 files(0 bytes)
> WARN  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for following
error
> java.lang.RuntimeException: Unknown column added_time during deserialization
>         at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
~[main/:na]
>         at org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
~[main/:na]
>         at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
~[main/:na]
>         at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
[main/:na]
>         at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
[main/:na]
>         at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
[main/:na]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[main/:na]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
>         at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
~[main/:na]
>         at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
~[main/:na]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[main/:na]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:55 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with /54.210.187.114
is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare completed.
Receiving 38 files(2323537628 bytes), sending 0 files(0 bytes)
> WARN  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for following
error
> java.lang.RuntimeException: Unknown column added_time during deserialization
>         at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
~[main/:na]
>         at org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
~[main/:na]
>         at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
~[main/:na]
>         at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
[main/:na]
>         at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
[main/:na]
>         at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
[main/:na]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[main/:na]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
>         at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
~[main/:na]
>         at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
~[main/:na]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[main/:na]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with /54.210.184.198
is complete
> INFO  18:45:56 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a ID#0] Prepare completed.
Receiving 35 files(2069893759 bytes), sending 0 files(0 bytes)
> WARN  18:45:57 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Retrying for following
error
> java.lang.AssertionError: null
>         at org.apache.cassandra.db.PartitionColumns$Builder.add(PartitionColumns.java:168)
~[main/:na]
>         at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:333)
~[main/:na]
>         at org.apache.cassandra.streaming.StreamReader.createWriter(StreamReader.java:136)
~[main/:na]
>         at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:77)
~[main/:na]
>         at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
[main/:na]
>         at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
[main/:na]
>         at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
[main/:na]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[main/:na]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> ERROR 18:45:57 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Streaming error occurred
> java.lang.IllegalArgumentException: Unknown type 0
>         at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
~[main/:na]
>         at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
~[main/:na]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
~[main/:na]
>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
> INFO  18:45:57 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Session with /54.210.157.2
is complete
> WARN  18:45:57 [Stream #9f95fa90-7691-11e5-931f-5b735851f84a] Stream failed
> ERROR 18:45:57 Error while waiting on bootstrap to complete. Bootstrap will have to be
restarted.
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
>         at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1245)
[main/:na]
>         at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:935)
[main/:na]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:710)
[main/:na]
>         at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:345)
[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:561)
[main/:na]
>         at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
[main/:na]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>         at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[main/:na]
>         at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
~[main/:na]
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
~[main/:na]
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
~[main/:na]
>         at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:525)
~[main/:na]
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279)
~[main/:na]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]
> Oct 19, 2015 6:45:57 PM com.google.common.util.concurrent.ExecutionList executeListener
> SEVERE: RuntimeException while executing runnable com.google.common.util.concurrent.Futures$6@59444306
with executor INSTANCE
> java.lang.NullPointerException
>         at org.apache.cassandra.service.StorageService$2.onFailure(StorageService.java:1240)
>         at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>         at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
>         at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
>         at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
>         at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
>         at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
>         at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
>         at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
>         at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:525)
>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:279)
>         at java.lang.Thread.run(Thread.java:745)
> {code}
> The "added_time" is a bigint column. I do see that the node is marked as UJ in nodetool
status and the bootstraping continue to run. But at a certain point, it hangs forever  and
restarting it doesn't help. I'm not 100% sure the hang issue is related to the above error
though. But I'd like to fix this one before trying again. Any idea what's going on with that
bigint type?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message