cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ingram Chen <ingramc...@gmail.com>
Subject Re: tcp CLOSE_WAIT bug
Date Tue, 20 Apr 2010 14:33:31 GMT
I trace IncomingStreamReader source and found that incoming socket comes
from MessagingService$SocketThread.
but there is no close() call on either accepted socket or socketChannel.

Should I file a bug report ?

On Tue, Apr 20, 2010 at 11:02, Ingram Chen <ingramchen@gmail.com> wrote:

> this happened after several hours of operations and both nodes are started
> at the same time (clean start without any data). so it might not relate to
> Bootstrap.
>
> In system.log I do not see any logs like "xxx node dead" or exceptions. and
> both nodes in test are alive. they serve read/write well, too. Below four
> connections between nodes are keep healthy from time to time.
>
>
> tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:58447
> ESTABLISHED
> tcp        0      0 ::ffff:192.168.2.87:54986   ::ffff:192.168.2.88:7000
> ESTABLISHED
> tcp        0      0 ::ffff:192.168.2.87:59138   ::ffff:192.168.2.88:7000
> ESTABLISHED
> tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:39074
> ESTABLISHED
>
> so connections end in CLOSE_WAIT should be newly created. (for streaming ?)
> This seems related to streaming issues we suffered recently:
> http://n2.nabble.com/busy-thread-on-IncomingStreamReader-td4908640.html
>
> I would like add some debug codes around opening and closing of socket to
> find out what happend.
>
> Could you give me some hint, about what classes I should take look ?
>
>
>
> On Tue, Apr 20, 2010 at 04:47, Jonathan Ellis <jbellis@gmail.com> wrote:
>
>> Is this after doing a bootstrap or other streaming operation?  Or did
>> a node go down?
>>
>> The internal sockets are supposed to remain open, otherwise.
>>
>> On Mon, Apr 19, 2010 at 10:56 AM, Ingram Chen <ingramchen@gmail.com>
>> wrote:
>> > Thank your information.
>> >
>> > We do use connection pools with thrift client and ThriftAdress is on
>> port
>> > 9160.
>> >
>> > Those problematic connections we found are all in port 7000, which is
>> > internal communications port between
>> > nodes. I guess this related to StreamingService.
>> >
>> > On Mon, Apr 19, 2010 at 23:46, Brandon Williams <driftx@gmail.com>
>> wrote:
>> >>
>> >> On Mon, Apr 19, 2010 at 10:27 AM, Ingram Chen <ingramchen@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>>     We have observed several connections between nodes in CLOSE_WAIT
>> >>> after several hours of operation:
>> >>
>> >> This is symptomatic of not pooling your client connections correctly.
>>  Be
>> >> sure you're using one connection per thread, not one connection per
>> >> operation.
>> >> -Brandon
>> >
>> >
>> > --
>> > Ingram Chen
>> > online share order: http://dinbendon.net
>> > blog: http://www.javaworld.com.tw/roller/page/ingramchen
>> >
>>
>
>
>
> --
> Ingram Chen
> online share order: http://dinbendon.net
> blog: http://www.javaworld.com.tw/roller/page/ingramchen
>



-- 
Ingram Chen
online share order: http://dinbendon.net
blog: http://www.javaworld.com.tw/roller/page/ingramchen

Mime
View raw message