cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aleksey Yeschenko (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CASSANDRA-8898) sstableloader utility should allow loading of data from mounted filesystem
Date Wed, 24 Feb 2016 15:23:18 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-8898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aleksey Yeschenko resolved CASSANDRA-8898.
------------------------------------------
    Resolution: Cannot Reproduce

We've had many changes to that code in the last few releases, and it's likely have been resolved
since then. If you can reproduce in latest 2.2.x or 3.0.x, please let us know, and reopen
the JIRA.

> sstableloader utility should allow loading of data from mounted filesystem
> --------------------------------------------------------------------------
>
>                 Key: CASSANDRA-8898
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8898
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>         Environment: 2.0.12
>            Reporter: Kenneth Failbus
>
> When trying to load data from a mounted filesystem onto a new cluster, following exceptions
is observed intermittently, and at some point the sstableloader process gets hung without
completing the loading process.
> Please note that in my case the scenario was loading the existing sstables from an existing
cluster to a brand new cluster.
> Finally, it was found that there were some hard assumptions been made by sstableloader
utility w.r.t response from the filesystem, which was not working with mounted filesystem.
> The work-around was to copy each existing nodes sstable data files locally and then point
sstableloader to that local filesystem to then load data onto new cluster.
> In case of restoring during disaster recovery from backups the data using sstableloader,
this copying to local filesystem of data files and then loading would take a long time.
> It would be a good enhancement of the sstableloader utility to enable use of mounted
filesystem as copying data locally and then loading is time consuming.
> Below is the exception seen during the use of mounted filesystem.
> {code}
> java.lang.AssertionError: Reference counter -1 for /opt/tmp/casapp-c1-c00053-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5449-Data.db

>         at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)

>         at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)

>         at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)

>         at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)

>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)

>         at java.lang.Thread.run(Thread.java:744) 
> WARN 21:07:16,853 [Stream #3e5a5ba0-bdef-11e4-a975-5777dbff0945] Stream failed
>   at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:59)

>         at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableReader.java:1406)

>         at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:55)

>         at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:59)

>         at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42)

>         at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:45)

>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:339)

>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:311)

>         at java.lang.Thread.run(Thread.java:744) 
> Caused by: java.io.FileNotFoundException: /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-5997-Data.db
(No such file or directory) 
>         at java.io.RandomAccessFile.open(Native Method) 
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)

>         at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:58)

>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:76)

>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:55)

>         ... 8 more 
> Exception in thread "STREAM-OUT-/96.115.88.196" java.lang.NullPointerException 
>         at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)

>         at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:331)

>         at java.lang.Thread.run(Thread.java:744) 
> ERROR 20:49:35,646 [Stream #d9fce650-bdf3-11e4-b6c0-252cb9b3e9f3] Streaming error occurred

> java.lang.AssertionError: Reference counter -3 for /opt/tmp/casapp-c1-c00055-g.ch.tvx.comcast.com/MinDataService/System/MinDataService-System-jb-4897-Data.db

>         at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1146)

>         at org.apache.cassandra.streaming.StreamTransferTask.complete(StreamTransferTask.java:74)

>         at org.apache.cassandra.streaming.StreamSession.received(StreamSession.java:542)

>         at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:424)

>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:245)

>         at java.lang.Thread.run(Thread.java:744) 
> Exception in thread "STREAM-IN-/96.115.88.196" java.lang.NullPointerException 
>         at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:205)

>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:260)

>         at java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message