cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Failbus (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CASSANDRA-8898) sstableloader utility should allow loading of data from mounted filesystem
Date Tue, 03 Mar 2015 18:01:05 GMT
Kenneth Failbus created CASSANDRA-8898:
------------------------------------------

             Summary: sstableloader utility should allow loading of data from mounted filesystem
                 Key: CASSANDRA-8898
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8898
             Project: Cassandra
          Issue Type: Improvement
          Components: Tools
         Environment: 2.0.12
            Reporter: Kenneth Failbus


When trying to load data from a mounted filesystem onto a new cluster, following exceptions
is observed intermittently, and at some point the sstableloader process gets hung without
completing the loading process.

Please note that in my case the scenario was loading the existing sstables from an existing
cluster to a brand new cluster.

Finally, it was found that there were some hard assumptions been made by sstableloader utility
w.r.t response from the filesystem, which was not working with mounted filesystem.

The work-around was to copy each existing nodes sstable data files locally and then point
sstableloader to that local filesystem to then load data.

In case of restoring during disaster recovery from backups the data using sstableloader, this
copying to local filesystem of data files and then loading would take a long time.

It would be a good enhancement of the sstableloader utility to enable use of mounted filesystem
as copying data locally and then loading is time consuming.

Below is the exception seen during the use of mounted filesystem.
{code}
java.lang.AssertionError: Reference counter -1 for /opt/tmp/casapp-c1-c00053-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5449-Data.db

        at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)

        at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)

        at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)

        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)

        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)

        at java.lang.Thread.run(Thread.java:744) 
WARN 21:07:16,853 [Stream #3e5a5ba0-bdef-11e4-a975-5777dbff0945] Stream failed

  at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59)

        at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1406)

        at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:55)

        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)

        at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)

        at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)

        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)

        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)

        at java.lang.Thread.run(Thread.java:744) 
Caused by: java.io.FileNotFoundException: /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5997-Data.db
(No such file or directory) 
        at java.io.RandomAccessFile.open(Native Method) 
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)

        at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)

        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)

        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)

        ... 8 more 
Exception in thread "STREAM-OUT-/96.115.88.196" java.lang.NullPointerException 
        at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)

        at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)

        at java.lang.Thread.run(Thread.java:744) 
ERROR 20:49:35,646 [Stream #d9fce650-bdf3-11e4-b6c0-252cb9b3e9f3] Streaming error occurred

java.lang.AssertionError: Reference counter -3 for /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-4897-Data.db

        at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)

        at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)

        at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)

        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)

        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)

        at java.lang.Thread.run(Thread.java:744) 
Exception in thread "STREAM-IN-/96.115.88.196" java.lang.NullPointerException 
        at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)

        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:260)

        at java.lang.Thread.run(Thread.java:744)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message