incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <tamara.alexan...@accenture.com>
Subject Decommissioning node is causing broken pipe error
Date Wed, 04 May 2011 00:29:28 GMT
Hi all,

I ran decommission on a node in my 32 node cluster. After about an hour of streaming files
to another node, I got this error on the node being decommissioned:
INFO [MiscStage:1] 2011-05-03 21:49:00,235 StreamReplyVerbHandler.java (line 58) Need to re-stream
file /raiddrive/MDR/MeterRecords-f-2283-Data.db to /10.206.63.208
ERROR [Streaming:1] 2011-05-03 21:49:01,580 DebuggableThreadPoolExecutor.java (line 103) Error
in ThreadPoolExecutor
java.lang.RuntimeException: java.io.IOException: Broken pipe
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
        at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:105)
        at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:67)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more
ERROR [Streaming:1] 2011-05-03 21:49:01,581 AbstractCassandraDaemon.java (line 112) Fatal
exception in thread Thread[Streaming:1,1,main]
java.lang.RuntimeException: java.io.IOException: Broken pipe
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Broken pipe
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:415)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:516)
        at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:105)
        at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:67)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more

And this message on the node that it was streaming to:
INFO [Thread-333] 2011-05-03 21:49:00,234 StreamInSession.java (line 121) Streaming of file
/raiddrive/MDR/MeterRecords-f-2283-Data.db/(98605680685,197932763967)
         progress=49016107008/99327083282 - 49% from org.apache.cassandra.streaming.StreamInSession@33721219
failed: requesting a retry.

I tried running decommission again (and running scrub + decommission), but I keep getting
this error on the same file.

I checked out the file and saw that it is a lot bigger than all the other sstables... 184GB
instead of about 74MB. I haven't run a major compaction for a bit, so I'm trying to stream
658 sstables.

I'm using Cassandra 0.7.4, I have two data directories (I know that's not good practice...),
and all my nodes are on Amazon EC2.

Any thoughts on what could be going on or how to prevent this?

Thanks!
Tamara



________________________________
This message is for the designated recipient only and may contain privileged, proprietary,
or otherwise private information. If you have received it in error, please notify the sender
immediately and delete the original. Any other use of the email by you is prohibited.

Mime
View raw message