Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of bburruss@real.com designates
 207.188.23.7 as permitted sender)
Message-ID: <4BD20D46.4090703@real.com>
Date: Fri, 23 Apr 2010 14:12:38 -0700
From: "B. Todd Burruss" <bburruss@real.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100411)
MIME-Version: 1.0
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: MESSAGE-STREAMING-POOL exception
References: <4BD20472.1070704@real.com>
 <x2he06563881004231341yacbef094u3129b8600d354fdb@mail.gmail.com>
 	<4BD20830.40006@real.com>
 <w2se06563881004231352t49fa0e6cn88c64763b6a4b7f0@mail.gmail.com>
In-Reply-To: <w2se06563881004231352t49fa0e6cn88c64763b6a4b7f0@mail.gmail.com>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 7bit
Old-Return-Path: bburruss@real.com

https://issues.apache.org/jira/browse/CASSANDRA-1019

Jonathan Ellis wrote:
> Can you create a ticket?
>
> On Fri, Apr 23, 2010 at 3:50 PM, B. Todd Burruss <bburruss@real.com> wrote:
>   
>> i agree, but it seems to have implications on the streaming service.
>>
>> Jonathan Ellis wrote:
>>     
>>> java.net.ConnectException: Connection timed out at
>>> sun.nio.ch.Net.connect is an os-level connection problem.
>>>
>>> On Fri, Apr 23, 2010 at 3:34 PM, B. Todd Burruss <bburruss@real.com>
>>> wrote:
>>>
>>>       
>>>> i see these exceptions on 4 out of the 7 nodes in my cluster.  in
>>>> addition
>>>> those same four nodes all show AE-SERVICE-STAGE with pending work, and
>>>> been
>>>> showing this for several hours now.  each node in the cluster has less
>>>> than
>>>> 2gb, so it should be finished by now.
>>>>
>>>> when i do nodetool streams on these nodes i see streams with byte counts
>>>> that are never increasing.
>>>>
>>>>
>>>> 2010-04-23 10:08:43,416 ERROR [MESSAGE-STREAMING-POOL:1]
>>>> [DebuggableThreadPoolExecutor.java:101] Error in ThreadPoolExecutor
>>>> java.lang.RuntimeException: java.net.ConnectException: Connection timed
>>>> out
>>>>      at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>>      at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>      at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>      at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.net.ConnectException: Connection timed out
>>>>      at sun.nio.ch.Net.connect(Native Method)
>>>>      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>>>>      at
>>>>
>>>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
>>>>      at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>      ... 3 more
>>>> 2010-04-23 10:08:43,417 ERROR [MESSAGE-STREAMING-POOL:1]
>>>> [CassandraDaemon.java:78] Fatal exception in thread
>>>> Thread[MESSAGE-STREAMING-POOL:1,5,main]
>>>> java.lang.RuntimeException: java.net.ConnectException: Connection timed
>>>> out
>>>>      at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>>      at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>      at
>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>      at java.lang.Thread.run(Thread.java:619)
>>>> Caused by: java.net.ConnectException: Connection timed out
>>>>      at sun.nio.ch.Net.connect(Native Method)
>>>>      at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507)
>>>>      at
>>>>
>>>> org.apache.cassandra.net.FileStreamTask.runMayThrow(FileStreamTask.java:60)
>>>>      at
>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>      ... 3 more
>>>>
>>>>
>>>>
>>>>