cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Problem while streaming SSTables with BulkOutputFormat
Date Tue, 09 Oct 2012 21:05:13 GMT
Something, somewhere, at some point is breaking the connection. Sorry I cannot be of more help
:)

Something caused the streaming to fail, which started a retry, which failed because the pipe
was broken. 

Are there any earlier errors in the logs ? 
Did this happen on one of the nodes that has both a task tacker and cassandra ?

Cheers


On 9/10/2012, at 4:06 AM, Ralph Romanos <matganine@hotmail.com> wrote:

> Hello,
> 
> I am using BulkOutputFormat to load data from a .csv file into Cassandra. I am using
Cassandra 1.1.3 and Hadoop 0.20.2.
> I have 7 hadoop nodes: 1 namenode/jobtracker and 6 datanodes/tasktrackers. Cassandra
is installed on 4 of these 6 datanodes/tasktrackers.
> The issue happens when I have more than 1 reducer, SSTables are generated in each node,
however, I get the following error in the tasktracker's logs when they 
> are streamed into the Cassandra cluster:
> 
> Exception in thread "Streaming to /172.16.110.79:1" java.lang.RuntimeException: java.io.EOFException
> 	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 	at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(Unknown Source)
> 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> 	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
> 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> 	... 3 more
> Exception in thread "Streaming to /172.16.110.92:1" java.lang.RuntimeException: java.io.EOFException
> 	at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> 	at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readInt(Unknown Source)
> 	at org.apache.cassandra.streaming.FileStreamTask.receiveReply(FileStreamTask.java:194)
> 	at org.apache.cassandra.streaming.FileStreamTask.stream(FileStreamTask.java:181)
> 	at org.apache.cassandra.streaming.FileStreamTask.runMayThrow(FileStreamTask.java:94)
> 	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> 	... 3 more
> 
>  ...
> 
> This is what I get in the logs of one of my Cassandra nodes:
> ERROR 16:47:34,904 Sending retry message failed, closing session.
> java.io.IOException: Broken pipe
>         at sun.nio.ch.FileDispatcher.write0(Native Method)
>         at sun.nio.ch.SocketDispatcher.write(Unknown Source)
>         at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
>         at sun.nio.ch.IOUtil.write(Unknown Source)
>         at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
>         at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
>         at java.nio.channels.Channels.writeFully(Unknown Source)
>         at java.nio.channels.Channels.access$000(Unknown Source)
>         at java.nio.channels.Channels$1.write(Unknown Source)
>         at java.io.OutputStream.write(Unknown Source)
>         at java.nio.channels.Channels$1.write(Unknown Source)
>         at java.io.DataOutputStream.writeInt(Unknown Source)
>         at org.apache.cassandra.net.OutboundTcpConnection.write(OutboundTcpConnection.java:196)
>         at org.apache.cassandra.streaming.StreamInSession.sendMessage(StreamInSession.java:171)
>         at org.apache.cassandra.streaming.StreamInSession.retry(StreamInSession.java:160)
>         at org.apache.cassandra.streaming.IncomingStreamReader.retry(IncomingStreamReader.java:168)
>         at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:98)
>         at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:182)
>         at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:78)
> 
> 
> Does anyone know what caused these errors?
> 
> Thank you for your help.
> Regards,
> Ralph



-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com


Mime
View raw message