Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of springrider@gmail.com
 designates 209.85.217.175 as permitted sender)
MIME-Version: 1.0
From: Yan Chunlu <springrider@gmail.com>
Date: Sun, 1 Jul 2012 23:14:58 +0800
Message-ID: 
 <CAOA66tGYS15e47kWbMk+mLwNhv4M9tBxDwzVNFEpS8v9RDXwDQ@mail.gmail.com>
Subject: cassandra halt after started minutes later
To: cassandra-user@incubator.apache.org
Content-Type: multipart/alternative; boundary=e89a8f23485535ecb604c3c626d1

--e89a8f23485535ecb604c3c626d1
Content-Type: text/plain; charset=ISO-8859-1

I have a three node cluster running 1.0.2, today there's a very strange
problem that suddenly two of cassandra  node(let's say B and C) was costing
a lot of cpu, turned out for some reason the "java" binary just dont
run.... I am using OpenJDK1.6.0_18, so I switched to "sun jdk", which works
okay.

after that node A stop working... same problem, I install "sun jdk", then
it's okay. but minutes later, B stop working again, about 5-10 minutes
later after the cassandra started, it stop responding connections, I can't
access 9160 and nodetool dont return either.

I have turned on DEBUG and dont see much useful information, the last rows
on node B are as belows:
DEBUG [pool-2-thread-72] 2012-07-01 07:45:42,830 RowDigestResolver.java
(line 65) resolving 2 responses
DEBUG [pool-2-thread-72] 2012-07-01 07:45:42,830 RowDigestResolver.java
(line 106) digests verified
DEBUG [pool-2-thread-72] 2012-07-01 07:45:42,830 RowDigestResolver.java
(line 110) resolve: 0 ms.
DEBUG [pool-2-thread-72] 2012-07-01 07:45:42,831 StorageProxy.java (line
694) Read: 5 ms.
DEBUG [Thread-8] 2012-07-01 07:45:42,831 IncomingTcpConnection.java (line
116) Version is now 3
DEBUG [Thread-8] 2012-07-01 07:45:42,831 IncomingTcpConnection.java (line
116) Version is now 3


this problem is really driving me crazy since I just dont know what
happened, and how to debug it, I tried to kill node A and restart it, then
node B halt, after I restart B, then node C goes down......


one thing may related is that the log time on node B is not the same with
the system time(A and C are okay).

while date on node B shows:
Sun Jul  1 23:10:57 CST 2012 (system time)

but you may noticed that the time is "2012-07-01 07:45:XX" in those above
log message.  the system time is right, just not sure why cassandra's log
file shows the wrong time, I didn't recall cassandra have timezone
settings.....

--e89a8f23485535ecb604c3c626d1
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have a three node cluster running 1.0.2, today there&#39;s a very strange=
 problem that suddenly two of cassandra =A0node(let&#39;s say B and C) was =
costing a lot of cpu, turned out for some reason the &quot;java&quot; binar=
y just dont run.... I am using OpenJDK1.6.0_18, so I switched to &quot;sun =
jdk&quot;, which works okay. <br>


<br>after that node A stop working... same problem, I install &quot;sun jdk=
&quot;, then it&#39;s okay. but minutes later, B stop working again, about =
5-10 minutes later after the cassandra started, it stop responding connecti=
ons, I can&#39;t access 9160 and nodetool dont return either. <br>


<br>I have turned on DEBUG and dont see much useful information, the last r=
ows on node B are as belows:<br>DEBUG [pool-2-thread-72] 2012-07-01 07:45:4=
2,830 RowDigestResolver.java (line 65) resolving 2 responses<br>DEBUG [pool=
-2-thread-72] 2012-07-01 07:45:42,830 RowDigestResolver.java (line 106) dig=
ests verified<br>


DEBUG [pool-2-thread-72] 2012-07-01 07:45:42,830 RowDigestResolver.java (li=
ne 110) resolve: 0 ms.<br>DEBUG [pool-2-thread-72] 2012-07-01 07:45:42,831 =
StorageProxy.java (line 694) Read: 5 ms.<br>DEBUG [Thread-8] 2012-07-01 07:=
45:42,831 IncomingTcpConnection.java (line 116) Version is now 3<br>


DEBUG [Thread-8] 2012-07-01 07:45:42,831 IncomingTcpConnection.java (line 1=
16) Version is now 3<br><br><br>this problem is really driving me crazy sin=
ce I just dont know what happened, and how to debug it, I tried to kill nod=
e A and restart it, then node B halt, after I restart B, then node C goes d=
own......<div>


<br></div><div><br></div><div>one thing may related is that the log time on=
 node B is not the same with the system time(A and C are okay).</div><div><=
br></div><div>while date on node B shows:</div><div><div>Sun Jul =A01 23:10=
:57 CST 2012 (system time)</div>

<div><br></div><div>but you may noticed that the time is &quot;2012-07-01 0=
7:45:XX&quot; in those above log message. =A0the system time is right, just=
 not sure why cassandra&#39;s log file shows the wrong time, I didn&#39;t r=
ecall cassandra have timezone settings.....</div>

<div><br></div><div><br></div>
</div><div><br></div><div><br></div>

--e89a8f23485535ecb604c3c626d1--