incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ingram Chen <ingramc...@gmail.com>
Subject tcp CLOSE_WAIT bug
Date Mon, 19 Apr 2010 15:27:25 GMT
Hi all,

    We have observed several connections between nodes in CLOSE_WAIT after
several hours of operation:

At node 87:

netstat -tn | grep 7000
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:57625
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:51541
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:58447
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:51313
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:52065
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:58218
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:54986   ::ffff:192.168.2.88:7000
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:48272
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:55433
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:59138   ::ffff:192.168.2.88:7000
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:39074
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:59088
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:34012
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:55806
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:42472
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.87:7000    ::ffff:192.168.2.88:45033
CLOSE_WAIT

At the other node: 88

netstat -tn | grep 7000
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:59138
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:46143
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:38202
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:55852
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:39208
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:55378
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:51061
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:44911
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:58447   ::ffff:192.168.2.87:7000
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:59614
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:35033
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:39074   ::ffff:192.168.2.87:7000
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:54986
ESTABLISHED
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:54772
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:39925
CLOSE_WAIT
tcp        0      0 ::ffff:192.168.2.88:7000    ::ffff:192.168.2.87:38124
CLOSE_WAIT

the setup only uses two nodes, replication factor = 2 with latest jdk 6u20
and cassandra 0.6.0

Afaik CLOSE_WAIT indicates there are opened sockets do not close properly.
Is anyone experience similar problem ? How do I do to find the root cause ?

Any help is appreciated.

Mime
View raw message