incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cfavero <chris.fav...@tricast.com>
Subject sstableloader hangs at progress. never finishes
Date Fri, 20 Sep 2013 17:03:02 GMT
Hello,

I have tried unsuccessfully to stream snapshots from a 4 node cluster to a 2
node cluster. I have set up a different machine to run sstableloader on and
I can see in the logs that it starts the stream and that the tmp files are
created in the correct columnfamily folders.

we are on 1.2.5 of datastax community
destination server is 10.10.104.80 and 81
sstableloader server is 10.10.105.190

*the issue is that the command:*
sstableloader -v -t 1 --debug -d 10.10.104.80,10.10.104.81
/root/sstables/keyspace/ColFamily 

*Outputs the following and never completes.*

[user@sstableloader user]# sstableloader -v -t 1 --debug -d
10.10.104.80,10.10.104.81 /user/sstables/keyspace/columfamily

Streaming revelant part of
/user/sstables/keyspace/columfamily/keysp-columnfam-ic-69-Data.db
/user/sstables/keyspace/columfamily/keysp-columnfam-ic-70-Data.db to
[/10.10.104.81, /10.10.104.80]

progress: [/10.10.104.81 1/2 (35)] [/10.10.104.80 1/2 (35)] [total: 35 -
0MB/s (avg: 0MB/s)]

*the destination server logs show the following:*

*when streaming starts:*
DEBUG [Thread-50] 2013-09-20 11:55:49,661 IncomingTcpConnection.java (line
75) Connection version 6 from /10.10.105.190
DEBUG [Thread-50] 2013-09-20 11:55:49,662 StreamInSession.java (line 104)
Adding file
/user/sstables/keyspace/columnfam/keyspace-columnfam-ic-70-Data.db to Stream
Request queue
DEBUG [Thread-50] 2013-09-20 11:55:49,662 StreamInSession.java (line 104)
Adding file
/user/sstables/keyspace/columnfam/keyspace-columnfam-ic-69-Data.db to Stream
Request queue
DEBUG [Thread-50] 2013-09-20 11:55:49,662 IncomingStreamReader.java (line
112) Receiving stream
DEBUG [Thread-50] 2013-09-20 11:55:49,662 IncomingStreamReader.java (line
113) Creating file for
/var/lib/cassandra/data/keyspace/columnfam/keyspace-columnfam-tmp-ic-7-Data.db
with 128 estimated keys
DEBUG [Thread-50] 2013-09-20 11:55:49,663 ColumnFamilyStore.java (line 863)
Checking for sstables overlapping []

*When I finally ctrl-c the hung process:*
DEBUG [Thread-50] 2013-09-20 11:55:54,195 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-Data.db
DEBUG [Thread-50] 2013-09-20 11:55:54,196 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-Index.db
DEBUG [Thread-50] 2013-09-20 11:55:54,196 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-Filter.db
DEBUG [Thread-50] 2013-09-20 11:55:54,196 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-TOC.txt
DEBUG [Thread-50] 2013-09-20 11:55:54,196 FileUtils.java (line 110) Deleting
keyspace-columnfam-tmp-ic-7-CompressionInfo.db
DEBUG [Thread-50] 2013-09-20 11:55:54,196 SSTable.java (line 154) Deleted
/var/lib/cassandra/data/keyspace/columnfam/keyspace-columnfam-tmp-ic-7
 INFO [Thread-50] 2013-09-20 11:55:54,197 StreamInSession.java (line 136)
Streaming of file
/user/sstables/keyspace/columnfam/keyspace-columnfam-ic-70-Data.db
sections=1 progress=0/1617 - 0% for
org.apache.cassandra.streaming.StreamInSession@672baef4 failed: requesting a
retry.
ERROR [Thread-50] 2013-09-20 11:55:54,197 StreamInSession.java (line 143)
Sending retry message failed, closing session.
java.io.IOException: Broken pipe
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:94)
        at sun.nio.ch.IOUtil.write(IOUtil.java:65)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
        at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
        at java.nio.channels.Channels.writeFully(Channels.java:98)
        at java.nio.channels.Channels.access$000(Channels.java:61)
        at java.nio.channels.Channels$1.write(Channels.java:174)
        at java.io.OutputStream.write(OutputStream.java:75)
        at java.nio.channels.Channels$1.write(Channels.java:155)
        at java.io.DataOutputStream.writeInt(DataOutputStream.java:198)
        at
org.apache.cassandra.streaming.StreamReply$FileStatusSerializer.serialize(StreamReply.java:75)
        at
org.apache.cassandra.streaming.StreamReply$FileStatusSerializer.serialize(StreamReply.java:69)
        at
org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:131)
        at
org.apache.cassandra.net.OutboundTcpConnection.write(OutboundTcpConnection.java:240)
        at
org.apache.cassandra.streaming.StreamInSession.sendMessage(StreamInSession.java:151)
        at
org.apache.cassandra.streaming.StreamInSession.retry(StreamInSession.java:139)
        at
org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:219)
        at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:127)
        at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:238)
        at
org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:178)
        at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
ERROR [Thread-50] 2013-09-20 11:55:54,198 FailureDetector.java (line 154)
unknown endpoint /10.10.105.190
DEBUG [Thread-50] 2013-09-20 11:55:54,198 AbstractStreamSession.java (line
74) closing with status false





--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/sstableloader-hangs-at-progress-never-finishes-tp7590464.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Mime
View raw message