cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ingram Chen <ingramc...@gmail.com>
Subject Re: tcp CLOSE_WAIT bug
Date Tue, 20 Apr 2010 03:02:13 GMT
this happened after several hours of operations and both nodes are started
at the same time (clean start without any data). so it might not relate to
Bootstrap.

In system.log I do not see any logs like "xxx node dead" or exceptions. and
both nodes in test are alive. they serve read/write well, too. Below four
connections between nodes are keep healthy from time to time.

tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:58447
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.87:54986   ::ffff:192.168.2.88:7000
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.87:59138   ::ffff:192.168.2.88:7000
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:39074
ESTABLISHED

so connections end in CLOSE_WAIT should be newly created. (for streaming ?)
This seems related to streaming issues we suffered recently:
http://n2.nabble.com/busy-thread-on-IncomingStreamReader-td4908640.html

I would like add some debug codes around opening and closing of socket to
find out what happend.

Could you give me some hint, about what classes I should take look ?


On Tue, Apr 20, 2010 at 04:47, Jonathan Ellis <jbellis@gmail.com> wrote:

> Is this after doing a bootstrap or other streaming operation?  Or did
> a node go down?
>
> The internal sockets are supposed to remain open, otherwise.
>
> On Mon, Apr 19, 2010 at 10:56 AM, Ingram Chen <ingramchen@gmail.com>
> wrote:
> > Thank your information.
> >
> > We do use connection pools with thrift client and ThriftAdress is on port
> > 9160.
> >
> > Those problematic connections we found are all in port 7000, which is
> > internal communications port between
> > nodes. I guess this related to StreamingService.
> >
> > On Mon, Apr 19, 2010 at 23:46, Brandon Williams <driftx@gmail.com>
> wrote:
> >>
> >> On Mon, Apr 19, 2010 at 10:27 AM, Ingram Chen <ingramchen@gmail.com>
> >> wrote:
> >>>
> >>> Hi all,
> >>>
> >>>     We have observed several connections between nodes in CLOSE_WAIT
> >>> after several hours of operation:
> >>
> >> This is symptomatic of not pooling your client connections correctly.
>  Be
> >> sure you're using one connection per thread, not one connection per
> >> operation.
> >> -Brandon
> >
> >
> > --
> > Ingram Chen
> > online share order: http://dinbendon.net
> > blog: http://www.javaworld.com.tw/roller/page/ingramchen
> >
>



-- 
Ingram Chen
online share order: http://dinbendon.net
blog: http://www.javaworld.com.tw/roller/page/ingramchen

Mime
View raw message