Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of dan.hendry.junk@gmail.com
 designates 209.85.216.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=sEqoZEXFzgL1zBQykb1TSXsxXhfbUjWHezHcyg5V3KRI8cr3poEX11546EoTrp8+kr
         /1G/IOh964LSqAsKyt9/bZAfLu5PFQu9N4k/U+l+RBUoip1xLLOMvd7ixYuiCCbhcrNm
         53mVY2uxZIHQp9ZJh/l3EkYLuBRo8kS834Ssk=
MIME-Version: 1.0
Date: Mon, 20 Dec 2010 13:48:58 -0500
Message-ID: <AANLkTinMjP_VaYUcVV7Gs7ExpxxVWW7wOYL5inVZDZjF@mail.gmail.com>
Subject: Severe Reliability Problems - 0.7 RC2
From: Dan Hendry <dan.hendry.junk@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/mixed; boundary=20cf30050df6045b240497dbf997

--20cf30050df6045b240497dbf997
Content-Type: multipart/alternative; boundary=20cf30050df6045b170497dbf995

--20cf30050df6045b170497dbf995
Content-Type: text/plain; charset=ISO-8859-1

I have been having severe and strange reliability problems within my
Cassandra cluster. This weekend, all four of my nodes were down at once.
Even now I am loosing one every few hours. I have attached output from all
the system monitoring commands I can think of.

What seems to happen is that the java process locks up and sits and has 100%
system cpu usage (but no user-CPU) (there are 8 cores so 100%=1/8 total
capacity). JMX freezes and the node effectively dies, but there is typically
nothing unusual in the Cassandra logs. About the only thing which seems to
be correlated is the flushing of memtables tables. One of the strangest
stats I am getting when in this state is memory paging: 3727168.00 pages
scanned/second (see sar -B output). Occasionally, if I leave the process
alone (~1 h) it recovers (maybe 1 in 5 times), otherwise the only way to
terminate the Cassandra process is with a kill -9. When this happens,
Cassandra memory usage (as reported by JMX before it dies) is
also reasonable (ex 6 GB out of 12 GB heap and 24 GB system).

This feels more like a system level problem than a Cassandra problem so I
have tried diversifying my cluster, one node runs Ubuntu 10.10, the other
three 10.04. One runs OpenJDK (1.6.0_20), the rest run Sun JDK (1.6.0_22).
Neither change seems be correlated with the problem. These are pretty much
stock ubuntu installs so nothing special on that front.

Now this has been a relatively sudden development and I can potentially
attribute it to a few things:
1. Upgrading to RC2
2. Ever increasing amounts of data (there is less than 100 gb per node so
this should not be the problem).
3. Migrating from a set of machines where data+commit log directories were
on four small raid 5 hds to machines with two 500 gig drives: one for data
and one for commitlog + os. I have seen more IO wait on these new machines.
But they have the same memory and system settings.

I am about at my wits end on this one, any help would be appreciated.

--20cf30050df6045b170497dbf995
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I have been having severe and strange reliability problems within my Cassan=
dra cluster. This weekend, all four of my nodes were down at once. Even now=
 I am loosing one every few hours. I have attached output from all the syst=
em monitoring commands I can think of.<div>
<br></div><div>What seems to happen is that the java process locks up and s=
its and has 100% system cpu usage (but no user-CPU) (there are 8 cores so 1=
00%=3D1/8 total capacity). JMX freezes and the node effectively dies, but t=
here is typically nothing unusual in the Cassandra logs. About the only thi=
ng which seems to be correlated is the flushing of memtables tables. One of=
 the strangest stats I am getting when in this state is memory paging: 3727=
168.00 pages scanned/second (see sar -B output).=A0Occasionally, if I leave=
 the process alone (~1 h) it recovers (maybe 1 in 5 times), otherwise the o=
nly way to terminate the Cassandra process is with a kill -9. When this hap=
pens, Cassandra memory usage (as reported by JMX before it dies) is also=A0=
reasonable=A0(ex 6 GB out of 12 GB heap and 24 GB system).=A0</div>
<div><br></div><div>This feels more like a system level problem than a Cass=
andra problem so I have tried diversifying my cluster, one node runs Ubuntu=
 10.10, the other three 10.04. One runs OpenJDK (1.6.0_20), the rest run Su=
n JDK (1.6.0_22). Neither change seems be correlated with the problem. Thes=
e are pretty much stock ubuntu installs so nothing special on that front.</=
div>
<div><br></div><div>Now this has been a relatively sudden development and I=
 can potentially attribute it to a few things:</div><div>1. Upgrading to RC=
2=A0</div><div>2. Ever increasing amounts of data (there is less than 100 g=
b per node so this should not be the problem).</div>
<div>3. Migrating from a set of machines where data+commit log directories =
were on four small raid 5 hds to machines with two 500 gig drives: one for =
data and one for commitlog + os. I have seen more IO wait on these new mach=
ines. But they have the same memory and system settings.</div>
<div><br></div><div>I am about at my wits end on this one, any help would b=
e appreciated.=A0</div>

--20cf30050df6045b170497dbf995--

--20cf30050df6045b240497dbf997
Content-Type: text/plain; charset=US-ASCII; name="vmstat.txt"
Content-Disposition: attachment; filename="vmstat.txt"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ghxnodfn0

CnZtc3RhdCBvdXRwdXQKCnByb2NzIC0tLS0tLS0tLS0tbWVtb3J5LS0tLS0tLS0tLSAtLS1zd2Fw
LS0gLS0tLS1pby0tLS0gLXN5c3RlbS0tIC0tLS1jcHUtLS0tCiByICBiICAgc3dwZCAgIGZyZWUg
ICBidWZmICBjYWNoZSAgIHNpICAgc28gICAgYmkgICAgYm8gICBpbiAgIGNzIHVzIHN5IGlkIHdh
CiAxICAwICAgNTQ3MiAgOTMwNTYgICA4NTI4IDEwNzQ4NDM2ICAgIDAgICAgMCAgICA5MCAgICA3
NCAgIDg4ICAgMTEgIDMgIDIgOTQgIDIKIDEgIDAgICA1NDcyICA5MzA1MiAgIDg1MjggMTA3NDg0
NjAgICAgMCAgICAwICAgICAwICAgICA0ICA0MDkgIDUyMSAgMCAxOCA4MiAgMAogMSAgMCAgIDU0
NzIgIDkzMDUyICAgODUyOCAxMDc0ODQ2MCAgICAwICAgIDAgICAgIDAgICAgIDAgIDM2NSAgNDg1
ICAwIDE2IDg0ICAwCiAxICAwICAgNTQ3MiAgOTMwNTIgICA4NTI4IDEwNzQ4NDYwICAgIDAgICAg
MCAgICAgMCAgICAgMCAgMzU4ICA0ODEgIDAgMTEgODkgIDAKIDEgIDAgICA1NDcyICA5MzA1MiAg
IDg1MjggMTA3NDg0NjAgICAgMCAgICAwICAgICAwICAgICAwICAzNjkgIDQ4NCAgMCAxMiA4OCAg
MAoK
--20cf30050df6045b240497dbf997
Content-Type: text/plain; charset=US-ASCII; name="top.txt"
Content-Disposition: attachment; filename="top.txt"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ghxnoniq1

ClRPUCBPdXRwdXQgCgp0b3AgLSAxMDo1ODoxOSB1cCAxMjo1NywgIDIgdXNlcnMsICBsb2FkIGF2
ZXJhZ2U6IDEuMDcsIDEuMjEsIDEuNTEKVGFza3M6IDE4NSB0b3RhbCwgICAxIHJ1bm5pbmcsIDE4
NCBzbGVlcGluZywgICAwIHN0b3BwZWQsICAgMCB6b21iaWUKQ3B1KHMpOiAgMC4wJXVzLCAxMS4x
JXN5LCAgMC4wJW5pLCA4OC44JWlkLCAgMC4wJXdhLCAgMC4wJWhpLCAgMC4wJXNpLCAgMC4wJXN0
Ck1lbTogIDI0NzMzMzM2ayB0b3RhbCwgMjQ2NDMzODBrIHVzZWQsICAgIDg5OTU2ayBmcmVlLCAg
ICAgODc1MmsgYnVmZmVycwpTd2FwOiAgMTk1Mjc2MGsgdG90YWwsICAgICA1NDcyayB1c2VkLCAg
MTk0NzI4OGsgZnJlZSwgMTA3NDg2MTZrIGNhY2hlZAoKICBQSUQgVVNFUiAgICAgIFBSICBOSSAg
VklSVCAgUkVTICBTSFIgUyAlQ1BVICVNRU0gICAgVElNRSsgIENPTU1BTkQgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogMTQ0MiBkaGVuZHJ5ICAgMjAg
ICAwIDkxLjlnICAyMGcgOC40ZyBTICAxMDAgODguNCAyNjM6MTUuNzMgamF2YSAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAxMTAxIHJvb3QgICAg
ICAyMCAgIDAgMTEyODAgIDU4NCAgNDQ4IFMgICAgMCAgMC4wICAgMDo0Ni40OCBpcnFiYWxhbmNl
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAKIDQzNTIgZGhl
bmRyeSAgIDIwICAgMCAxOTIyMCAxNDcyIDEwNjQgUiAgICAwICAwLjAgICAwOjAwLjAyIHRvcCAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIAogICAg
MSByb290ICAgICAgMjAgICAwIDIzNzA0IDE3NjQgMTIxNiBTICAgIDAgIDAuMCAgIDA6MDEuNTQg
aW5pdCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
CiAgICAyIHJvb3QgICAgICAyMCAgIDAgICAgIDAgICAgMCAgICAwIFMgICAgMCAgMC4wICAgMDow
MC4wMiBrdGhyZWFkZCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAKICAgIDMgcm9vdCAgICAgIFJUICAgMCAgICAgMCAgICAwICAgIDAgUyAgICAwICAwLjAg
ICAwOjAwLjAwIG1pZ3JhdGlvbi8wICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgIAoK
--20cf30050df6045b240497dbf997
Content-Type: text/plain; charset=US-ASCII; name="sar-u.txt"
Content-Disposition: attachment; filename="sar-u.txt"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ghxnos2i2

CgpvdXRwdXQgb2Ygc2FyIC11IDEgNQoKTGludXggMi42LjMyLTIxLXNlcnZlciAodGFsazEyKSAJ
MTIvMjAvMjAxMCAJX3g4Nl82NF8JKDggQ1BVKQoKMTE6MTI6MjUgQU0gICAgIENQVSAgICAgJXVz
ZXIgICAgICVuaWNlICAgJXN5c3RlbSAgICVpb3dhaXQgICAgJXN0ZWFsICAgICAlaWRsZQoxMTox
MjoyNiBBTSAgICAgYWxsICAgICAgMC4wMCAgICAgIDAuMDAgICAgIDE4LjIwICAgICAgMC4wMCAg
ICAgIDAuMDAgICAgIDgxLjgwCjExOjEyOjI3IEFNICAgICBhbGwgICAgICAwLjAwICAgICAgMC4w
MCAgICAgMTMuNzAgICAgICAwLjQxICAgICAgMC4wMCAgICAgODUuODkKMTE6MTI6MjggQU0gICAg
IGFsbCAgICAgIDAuMDAgICAgICAwLjAwICAgICAxMC41MiAgICAgIDAuMDAgICAgICAwLjAwICAg
ICA4OS40OAoxMToxMjoyOSBBTSAgICAgYWxsICAgICAgMC4wMCAgICAgIDAuMDAgICAgIDE2Ljk0
ICAgICAgMC4wMCAgICAgIDAuMDAgICAgIDgzLjA2CjExOjEyOjMwIEFNICAgICBhbGwgICAgICAw
LjAwICAgICAgMC4wMCAgICAgMTQuOTIgICAgICAwLjAwICAgICAgMC4wMCAgICAgODUuMDgKQXZl
cmFnZTogICAgICAgIGFsbCAgICAgIDAuMDAgICAgICAwLjAwICAgICAxNC4zMiAgICAgIDAuMDgg
ICAgICAwLjAwICAgICA4NS42MAoK
--20cf30050df6045b240497dbf997
Content-Type: text/plain; charset=US-ASCII; name="sar-B.txt"
Content-Disposition: attachment; filename="sar-B.txt"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ghxnozi43

Ck91dHB1dCBvZiBzYXIgLUIgMSA1CgpMaW51eCAyLjYuMzItMjEtc2VydmVyICh0YWxrMTIpIAkx
Mi8yMC8yMDEwIAlfeDg2XzY0XwkoOCBDUFUpCgoxMToxMjo1OSBBTSAgcGdwZ2luL3MgcGdwZ291
dC9zICAgZmF1bHQvcyAgbWFqZmx0L3MgIHBnZnJlZS9zIHBnc2NhbmsvcyBwZ3NjYW5kL3MgcGdz
dGVhbC9zICAgICV2bWVmZgoxMToxMzowMCBBTSAgICAgIDAuMDAgICAgICAwLjAwICAgICAzOC4w
MCAgICAgIDAuMDAgICAgIDYxLjAwICAgICAgMC4wMCAzNzI3MTY4LjAwICAgICAgMC4wMCAgICAg
IDAuMDAKMTE6MTM6MDEgQU0gICAgICAwLjAwICAgICAgMC4wMCAgICAgMzkuMDAgICAgICAwLjAw
ICAgICA2MS4wMCAgICAgIDAuMDAgMzcxNDQwMC4wMCAgICAgIDAuMDAgICAgICAwLjAwCjExOjEz
OjAyIEFNICAgICAgMC4wMCAgICAgIDAuMDAgICAgIDMzLjAwICAgICAgMC4wMCAgICAgNjEuMDAg
ICAgICAwLjAwIDM3MjQzODQuMDAgICAgICAwLjAwICAgICAgMC4wMAoxMToxMzowMyBBTSAgICAg
IDAuMDAgICAgICAwLjAwICAgICAzMy4wMCAgICAgIDAuMDAgICAgIDYxLjAwICAgICAgMC4wMCAz
NzE2MTkyLjAwICAgICAgMC4wMCAgICAgIDAuMDAKMTE6MTM6MDQgQU0gICAgICAwLjAwICAgICAg
MC4wMCAgICAgMzMuMDAgICAgICAwLjAwICAgICA2MS4wMCAgICAgIDAuMDAgMzcxMzE4NC4wMCAg
ICAgIDAuMDAgICAgICAwLjAwCkF2ZXJhZ2U6ICAgICAgICAgMC4wMCAgICAgIDAuMDAgICAgIDM1
LjIwICAgICAgMC4wMCAgICAgNjEuMDAgICAgICAwLjAwIDM3MTkwNjUuNjAgICAgICAwLjAwICAg
ICAgMC4wMAoK
--20cf30050df6045b240497dbf997
Content-Type: text/plain; charset=US-ASCII; name="iostat.txt"
Content-Disposition: attachment; filename="iostat.txt"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ghxnqhuo4

Cmlvc3RhdCBvdXRwdXQ6CgphdmctY3B1OiAgJXVzZXIgICAlbmljZSAlc3lzdGVtICVpb3dhaXQg
ICVzdGVhbCAgICVpZGxlCiAgICAgICAgICAgMi4zMCAgICAwLjM5ICAgIDEuOTUgICAgMS44MyAg
ICAwLjAwICAgOTMuNTQKCkRldmljZTogICAgICAgICAgICB0cHMgICBCbGtfcmVhZC9zICAgQmxr
X3dydG4vcyAgIEJsa19yZWFkICAgQmxrX3dydG4Kc2RhICAgICAgICAgICAgICAgMi4wMyAgICAg
ICAgMjYuMDQgICAgICAgNDk2LjczICAgIDEyMTY2MjIgICAyMzIwNzYxMApzZGIgICAgICAgICAg
ICAgIDIxLjU5ICAgICAgMTQyNS43MCAgICAgICA2OTcuMzMgICA2NjYwOTcyMiAgIDMyNTc5ODE2
CgphdmctY3B1OiAgJXVzZXIgICAlbmljZSAlc3lzdGVtICVpb3dhaXQgICVzdGVhbCAgICVpZGxl
CiAgICAgICAgICAgMC4wMCAgICAwLjAwICAgMTYuNTMgICAgMC4wMCAgICAwLjAwICAgODMuNDcK
CkRldmljZTogICAgICAgICAgICB0cHMgICBCbGtfcmVhZC9zICAgQmxrX3dydG4vcyAgIEJsa19y
ZWFkICAgQmxrX3dydG4Kc2RhICAgICAgICAgICAgICAgMC4wMCAgICAgICAgIDAuMDAgICAgICAg
ICAwLjAwICAgICAgICAgIDAgICAgICAgICAgMApzZGIgICAgICAgICAgICAgICAwLjAwICAgICAg
ICAgMC4wMCAgICAgICAgIDAuMDAgICAgICAgICAgMCAgICAgICAgICAwCgphdmctY3B1OiAgJXVz
ZXIgICAlbmljZSAlc3lzdGVtICVpb3dhaXQgICVzdGVhbCAgICVpZGxlCiAgICAgICAgICAgMC4w
MCAgICAwLjAwICAgMTAuNTkgICAgMC4wMCAgICAwLjAwICAgODkuNDEKCkRldmljZTogICAgICAg
ICAgICB0cHMgICBCbGtfcmVhZC9zICAgQmxrX3dydG4vcyAgIEJsa19yZWFkICAgQmxrX3dydG4K
c2RhICAgICAgICAgICAgICAgMC4wMCAgICAgICAgIDAuMDAgICAgICAgICAwLjAwICAgICAgICAg
IDAgICAgICAgICAgMApzZGIgICAgICAgICAgICAgICAwLjAwICAgICAgICAgMC4wMCAgICAgICAg
IDAuMDAgICAgICAgICAgMCAgICAgICAgICAwCgo=
--20cf30050df6045b240497dbf997
Content-Type: text/plain; charset=US-ASCII; name="free.txt"
Content-Disposition: attachment; filename="free.txt"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ghxnqrrk5

CgpvdXRwdXQgb2YgZnJlZSAtbQoKICAgICAgICAgICAgIHRvdGFsICAgICAgIHVzZWQgICAgICAg
ZnJlZSAgICAgc2hhcmVkICAgIGJ1ZmZlcnMgICAgIGNhY2hlZApNZW06ICAgICAgICAgMjQxNTMg
ICAgICAyNDAzOSAgICAgICAgMTE0ICAgICAgICAgIDAgICAgICAgICAgOSAgICAgIDEwNDU5Ci0v
KyBidWZmZXJzL2NhY2hlOiAgICAgIDEzNTcwICAgICAgMTA1ODIKU3dhcDogICAgICAgICAxOTA2
ICAgICAgICAgIDUgICAgICAgMTkwMQoK
--20cf30050df6045b240497dbf997
Content-Type: text/plain; charset=US-ASCII; name="cassandra.txt"
Content-Disposition: attachment; filename="cassandra.txt"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_ghxnqyt46

CkNhc3NhbmRyYSBsb2c6CgogSU5GTyBbQ09NTUlULUxPRy1XUklURVJdIDIwMTAtMTItMjAgMTA6
NDQ6NDEsOTM2IENvbW1pdExvZ1NlZ21lbnQuamF2YSAobGluZSA1MCkgQ3JlYXRpbmcgbmV3IGNv
bW1pdGxvZyBzZWdtZW50IC92YXIvbGliL2Nhc3NhbmRyYS9jb21taXRsb2cvQ29tbWl0TG9nLTEy
OTI4NjM0ODE5MzYubG9nCiBJTkZPIFtDT01NSVQtTE9HLVdSSVRFUl0gMjAxMC0xMi0yMCAxMDo0
NTozOSwxNjcgQ29tbWl0TG9nU2VnbWVudC5qYXZhIChsaW5lIDUwKSBDcmVhdGluZyBuZXcgY29t
bWl0bG9nIHNlZ21lbnQgL3Zhci9saWIvY2Fzc2FuZHJhL2NvbW1pdGxvZy9Db21taXRMb2ctMTI5
Mjg2MzUzOTE2Ni5sb2cKIElORk8gW011dGF0aW9uU3RhZ2U6MjZdIDIwMTAtMTItMjAgMTA6NDg6
MjksMTQwIENvbHVtbkZhbWlseVN0b3JlLmphdmEgKGxpbmUgNjQwKSBzd2l0Y2hpbmcgaW4gYSBm
cmVzaCBNZW10YWJsZSBmb3IgVXNlckV2ZW50c0J5VXNlciBhdCBDb21taXRMb2dDb250ZXh0KGZp
bGU9Jy92YXIvbGliL2Nhc3NhbmRyYS9jb21taXRsb2cvQ29tbWl0TG9nLTEyOTI4NjM1MzkxNjYu
bG9nJywgcG9zaXRpb249NDk4MTI0MjEpCiBJTkZPIFtNdXRhdGlvblN0YWdlOjI2XSAyMDEwLTEy
LTIwIDEwOjQ4OjI5LDE0MCBDb2x1bW5GYW1pbHlTdG9yZS5qYXZhIChsaW5lIDk0NCkgRW5xdWV1
aW5nIGZsdXNoIG9mIE1lbXRhYmxlLVVzZXJFdmVudHNCeVVzZXJAMTc4NDM3MTc1NCgxNjI2MDYy
NjEgYnl0ZXMsIDIwOTc2MTMgb3BlcmF0aW9ucykKIElORk8gW0ZsdXNoV3JpdGVyOjFdIDIwMTAt
MTItMjAgMTA6NDg6MjksMTQwIE1lbXRhYmxlLmphdmEgKGxpbmUgMTU1KSBXcml0aW5nIE1lbXRh
YmxlLVVzZXJFdmVudHNCeVVzZXJAMTc4NDM3MTc1NCgxNjI2MDYyNjEgYnl0ZXMsIDIwOTc2MTMg
b3BlcmF0aW9ucykKCg==
--20cf30050df6045b240497dbf997--