cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Kania <jason.ka...@ymail.com>
Subject Re: How to complete bootstrap with exception due to stream failure?
Date Mon, 29 Feb 2016 03:25:58 GMT
Thanks for the reference to nodetool resetlocalschema as that will come in handy in the future.
Thanks also for the reference to https://issues.apache.org/jira/browse/CASSANDRA-11050 which
seems related, but I am not sure.

I was doing a bootstrapping on 192.168.10.10 and it had nothing on it to start with it. It
was in the process of transferring the schema definitions that the bootstrap was failing.
In the process of trying to get something working, I tried adding the dropped columns on the
existing node and the new node but had no luck with that either.
I finally figured it out so I raised https://issues.apache.org/jira/browse/CASSANDRA-11273
with these details and the workaround that I found.
      From: Paulo Motta <pauloricardomg@gmail.com>
 To: "user@cassandra.apache.org" <user@cassandra.apache.org>; Jason Kania <jason.kania@ymail.com>

 Sent: Sunday, February 28, 2016 10:01 PM
 Subject: Re: How to complete bootstrap with exception due to stream failure?
   
Were the columns sensor.lastEvaluation and sensordb.lastCheckTime dropped by any chance? If
so, you might be hitting https://issues.apache.org/jira/browse/CASSANDRA-11050, fixed in upcoming
3.4.

If that's the case, you may want to check if nodes other than 192.168.10.10 have the dropped
columns in the system_schema.dropped_columns table, and if so, reset the local schema (nodetool
resetlocalschema) of 192.168.10.10 to force a schema synchronization with other nodes. Another
possible workaround is to manually include the dropped columns in the system_schema.dropped_columns
table of 192.168.10.10.

2016-02-27 22:56 GMT-03:00 Jason Kania <jason.kania@ymail.com>:

Hi,
I just reran the command and collected following. Any suggestions would be appreciated.

Thanks,
Jason

from 192.168.10.8

ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10 failed stream session.
INFO  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamResultFuture.java:182 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Session with /192.168.10.10 is complete
WARN  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 StreamResultFuture.java:209 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Stream failed

from 192.168.10.8 debug
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,414 ConnectionHandler.java:262 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Received Received (79256340-bbbb-11e5-9f70-7d76a8de8480,
#0)
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,854 ConnectionHandler.java:262 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Received Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7,
#0)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 ConnectionHandler.java:334 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Sending File (Header (cfId: f3a137e0-024b-11e5-bb31-0d2316086bf7,
#0, version: ma, format: BIG, estimated keys: 128, transfer size: 4653, compressed?: true,
repairedAt: 0, level: 0), file: /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 CompressedStreamWriter.java:63 -
[Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db
to /192.168.10.10, repairedAt = 0, totalSize = 4653
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,854 CompressedStreamWriter.java:94 -
[Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file /home/cassandra/data/sensordb/sensor/ma-76-big-Data.db
to /192.168.10.10, bytesTransferred = 4653, totalSize = 4653
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,855 ConnectionHandler.java:262 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Received Retry (faa55490-024b-11e5-bb31-0d2316086bf7,
#0)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,855 ConnectionHandler.java:334 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Sending File (Header (cfId: faa55490-024b-11e5-bb31-0d2316086bf7,
#0, version: ma, format: BIG, estimated keys: 128, transfer size: 705, compressed?: true,
repairedAt: 0, level: 0), file: /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 CompressedStreamWriter.java:63 -
[Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Start streaming file /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db
to /192.168.10.10, repairedAt = 0, totalSize = 705
DEBUG [STREAM-OUT-/192.168.10.10] 2016-02-27 20:37:53,856 CompressedStreamWriter.java:94 -
[Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished streaming file /home/cassandra/data/sensordb/sensorUnit/ma-79-big-Data.db
to /192.168.10.10, bytesTransferred = 705, totalSize = 705
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 ConnectionHandler.java:262 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Received Session Failed
ERROR [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamSession.java:635 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Remote peer 192.168.10.10 failed stream session.
DEBUG [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 ConnectionHandler.java:110 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Closing stream connection handler on /192.168.10.10
INFO  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,857 StreamResultFuture.java:182 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Session with /192.168.10.10 is complete
WARN  [STREAM-IN-/192.168.10.10] 2016-02-27 20:37:53,858 StreamResultFuture.java:209 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Stream failed


from 192.168.10.10
[2016-02-27 20:37:53,413] received file /home/cassandra/data/sensordb/listedAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db
(progress: 365%)
[2016-02-27 20:37:53,414] received file /home/cassandra/data/sensordb/liestedAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db
(progress: 369%)
[2016-02-27 20:37:53,865] session with /192.168.10.8 complete (progress: 369%)
[2016-02-27 20:37:53,866] Stream failed

from 192.168.10.10 debug
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,201 CompressedStreamReader.java:80 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /192.168.10.8, repairedAt
= 0, size = 166627, ks = 'sensordb', table = 'listAttributes'.
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412 CompressedStreamReader.java:110 -
[Stream #c9868f90-ddbb-11e5-80c0-89f591237aca] Finished receiving file #0 from /192.168.10.8
readBytes = 166627, totalSize = 166627
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,412 ConnectionHandler.java:262 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Received File (Header (cfId: 79256340-bbbb-11e5-9f70-7d76a8de8480,
#0, version: ma, format: BIG, estimated keys: 128, transfer size: 166627, compressed?: true,
repairedAt: 0, level: 0), file: /home/cassandra/data/sensordb/listAttributes-79256340bbbb11e59f707d76a8de8480/ma-32-big-Data.db)
DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,412 ConnectionHandler.java:334 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Sending Received (79256340-bbbb-11e5-9f70-7d76a8de8480,
#0)
DEBUG [CompactionExecutor:3] 2016-02-27 20:37:53,833 CompactionTask.java:217 - Compacted (e224bef0-ddbb-11e5-80c0-89f591237aca)
4 sstables to [/home/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/ma-5-big,]
to level=0.  2,743,164 bytes to 685,791 (~25% of original) in 1,096ms = 0.596735MB/s.  0
total partitions merged to 57.  Partition merge counts were {4:57, }
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,850 CompressedStreamReader.java:80 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /192.168.10.8, repairedAt
= 0, size = 4653, ks = 'sensordb', table = 'sensor'.
WARN  [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,851 StreamSession.java:641 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Retrying for following error
java.lang.RuntimeException: Unknown column lastEvaluation during deserialization
        at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[apache-cassandra-3.0.3.jar:3.0.3]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
DEBUG [STREAM-OUT-/192.168.10.8] 2016-02-27 20:37:53,852 ConnectionHandler.java:334 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Sending Retry (f3a137e0-024b-11e5-bb31-0d2316086bf7,
#0)
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,852 ConnectionHandler.java:262 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Received null
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,853 CompressedStreamReader.java:80 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Start receiving file #0 from /192.168.10.8, repairedAt
= 0, size = 705, ks = 'sensordb', table = 'sensorUnit'.
WARN  [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854 StreamSession.java:641 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Retrying for following error
java.lang.RuntimeException: Unknown column lastCheckTime during deserialization
        at org.apache.cassandra.db.SerializationHeader$Component.toHeader(SerializationHeader.java:331)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:87)
~[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
[apache-cassandra-3.0.3.jar:3.0.3]
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
[apache-cassandra-3.0.3.jar:3.0.3]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
DEBUG [STREAM-IN-/192.168.10.8] 2016-02-27 20:37:53,854 ConnectionHandler.java:262 - [Stream
#c9868f90-ddbb-11e5-80c0-89f591237aca] Received null


      From: Sebastian Estevez <sebastian.estevez@datastax.com>
 To: user@cassandra.apache.org; Jason Kania <jason.kania@ymail.com> 
 Sent: Saturday, February 27, 2016 8:24 PM
 Subject: Re: How to complete bootstrap with exception due to stream failure?
  
progress: 361% does not look right (probably a bug).

Can you check the corresponding messages on the other side of the stream? I.E. the system
log for 192.168.10.8 around 18:02:06?
All the best,
Sebastián EstévezSolutions Architect | 954 905 8615 | sebastian.estevez@datastax.com   



DataStax is the fastest, most scalable distributed database technology, delivering Apache
Cassandra to the world’s most innovative enterprises. Datastax is built to be agile, always-on,
and predictably scalable to any size. With more than 500 customers in 45 countries, DataStax
is the database technology and transactional backbone of choice for the worlds most innovative
companies such as Netflix, Adobe, Intuit, and eBay. 
On Sat, Feb 27, 2016 at 6:12 PM, Jason Kania <jason.kania@ymail.com> wrote:

Hello,
I am trying to get a node bootstrapped in 3.0.3, but just at the point where the bootstrap
process is to complete, a broken pipe exception occurs so the bootstrap process hangs. Once
I kill the bootstrap process, I can execute "nodetool bootstrap resume" again and the same
problem will occur just at the end of the bootstrap exercise. Here is the tail of the log:
[2016-02-27 18:02:05,898] received file /home/cassandra/data/sensordb/listedAttributes-79256340bbbb11e59f707d76a8de8480/ma-30-big-Data.db
(progress: 357%)
[2016-02-27 18:02:06,479] received file /home/cassandra/data/sensordb/notification-f7e3eaa0024b11e5bb310d2316086bf7/ma-38-big-Data.db
(progress: 361%)
[2016-02-27 18:02:06,884] session with /192.168.10.8 complete (progress: 361%)
[2016-02-27 18:02:06,886] Stream failed
I attempted to run nodetool repair, but get the following which I have been told indicates
that the replication factor is 1:
root@bull:~# nodetool repair
[2016-02-27 18:04:55,083] Nothing to repair for keyspace 'sensordb'

Thanks,
Jason




   



  
Mime
View raw message