cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wateray (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10303) streaming for 'nodetool rebuild' fails after adding a datacenter
Date Thu, 03 Dec 2015 03:45:11 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037199#comment-15037199
] 

wateray commented on CASSANDRA-10303:
-------------------------------------

The node which execute rebuild, when it read stream throws java.io.IOException: Connection
timed out.,
It will hang up, because of this exception does not been  caughted.  The StreamSession didn't
know the exception, it cann't end up.

The latest release version 2.1.11,2.2.3 3.0 don't fix this bug. The develop trunk has fixed
this bug.
   CompressedInputStream.java 

*with bug version*
{code:java}
                while (bufferRead < readLength)
                {
                    int r = source.read(compressedWithCRC, bufferRead, readLength - bufferRead);
                    if (r < 0)
                    {
                        dataBuffer.put(POISON_PILL);
                        return; // throw exception where we consume dataBuffer
                    }
                    bufferRead += r;
                }
{code}


*fixed version*
{code:java}
        protected void runMayThrow() throws Exception
        {
..............
       while (bufferRead < readLength)
                {
                    int r;
                    try
                    {
                        r = source.read(compressedWithCRC, bufferRead, readLength - bufferRead);
                        if (r < 0)
                        {
                            dataBuffer.put(POISON_PILL);
                            return; // throw exception where we consume dataBuffer
                        }
                    }
                    catch (IOException e)
                    {
                        dataBuffer.put(POISON_PILL);
                        throw e;
                    }
                    bufferRead += r;
                }
                dataBuffer.put(compressedWithCRC);
            }

.......}
{code}

> streaming for 'nodetool rebuild' fails after adding a datacenter 
> -----------------------------------------------------------------
>
>                 Key: CASSANDRA-10303
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10303
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: jdk1.7
> cassandra 2.1.8
>            Reporter: zhaoyan
>             Fix For: 2.1.x
>
>
> we add another datacenter.
> use nodetool rebuild DC1
> stream from some node of old datacenter always hang up with these exception:
> {code}
> ERROR [Thread-1472] 2015-09-10 19:24:53,091 CassandraDaemon.java:223 - Exception in thread
Thread[Thread-1472,5,RMI Runtime]
> java.lang.RuntimeException: java.io.IOException: Connection timed out
>     at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32) ~[apache-cassandra-2.1.8.jar:2.1.8]
>     at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_60]
> Caused by: java.io.IOException: Connection timed out
>     at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.7.0_60]
>     at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.7.0_60]
>     at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.7.0_60]
>     at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.7.0_60]
>     at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) ~[na:1.7.0_60]
>     at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.7.0_60]
>     at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.7.0_60]
>     at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.7.0_60]
>     at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172)
~[apache-cassandra-2.1.8.jar:2.1.8]
>     at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.8.jar:2.1.8]
>     ... 1 common frames omitted
> {code}
> i must restart node to stop current rebuild, and rebuild agagin and again to success....



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message