Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of mike@simplegeo.com designates
 74.125.83.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BANLkTi=O3R8CEp8ARdYYbJtx5vbTnw3NYw@mail.gmail.com>
References: <BANLkTi=O3R8CEp8ARdYYbJtx5vbTnw3NYw@mail.gmail.com>
Date: Thu, 19 May 2011 10:00:48 -0700
Message-ID: <BANLkTimxMsnuxk=e5ENnBzy6M9KRcyVaVw@mail.gmail.com>
Subject: Re: Commitlog Disk Full
From: Mike Malone <mike@simplegeo.com>
To: user@cassandra.apache.org
Content-Type: multipart/mixed; boundary=000e0cd2e71669299904a3a3f250

--000e0cd2e71669299904a3a3f250
Content-Type: multipart/alternative; boundary=000e0cd2e71669297d04a3a3f24e

--000e0cd2e71669297d04a3a3f24e
Content-Type: text/plain; charset=ISO-8859-1

Just noticed this thread and figured I'd chime in since we've had similar
issues with the commit log growing too large on our clusters. Tuning down
the flush timeout wasn't really an acceptable solution for us since we
didn't want to be constantly flushing and generating extra SSTables for no
reason. So we wrote a small tool that we start in a static block in
CassandraServer that periodically checks the commit log size and flushes all
memtables if they're above some threshold.

I've attached that code. Any feedback / improvements are more than welcome.

Mike

On Thu, May 12, 2011 at 11:30 AM, Sanjeev Kulkarni <sanjeev@locomatix.com>wrote:

> Hey guys,
> I have a ec2 debian cluster consisting of several nodes running 0.7.5 on
> ephimeral disks.
> These are fresh installs and not upgrades.
> The commitlog is set to the smaller of the disks which is around 10G in
> size and the datadir is set to the bigger disk.
> The config file is basically the same as the one supplied by the default
> installation.
> Our applications write to the cluster. After about a day of writing we
> started noticing the commitlog disk filling up. Soon we went over the disk
> limit and writes started failing. At this point we stopped the cluster.
> Over the course of the day we inserted around 25G of data. Our columns
> values are pretty small.
> I understand that cassandra periodically cleans up the commitlog
> directories by generating sstables in datadir. Is there any way to speed up
> this movement from commitog to datadir?
> Thanks!
>
>

--000e0cd2e71669297d04a3a3f24e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Just noticed this thread and figured I&#39;d chime in since we&#39;ve had s=
imilar issues with the commit log growing too large on our clusters. Tuning=
 down the flush timeout wasn&#39;t really an acceptable solution for us sin=
ce we didn&#39;t want to be constantly flushing and generating extra SSTabl=
es for no reason. So we wrote a small tool that we start in a static block =
in CassandraServer that periodically checks the commit log size and flushes=
 all memtables if they&#39;re above some threshold.<div>
<div><br></div><div>I&#39;ve attached that code. Any feedback / improvement=
s are more than welcome.<br><div><br></div><div>Mike<br><br><div class=3D"g=
mail_quote">On Thu, May 12, 2011 at 11:30 AM, Sanjeev Kulkarni <span dir=3D=
"ltr">&lt;<a href=3D"mailto:sanjeev@locomatix.com">sanjeev@locomatix.com</a=
>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex;">Hey guys,<div>I have a ec2 debian cluster c=
onsisting of several nodes running 0.7.5 on ephimeral disks.=A0</div><div>T=
hese are fresh installs and not upgrades.=A0</div>
<div>The commitlog is set to the smaller of the disks which is around 10G i=
n size and the datadir is set to the bigger disk.</div>
<div>The config file is basically the same as the one supplied by the defau=
lt installation.</div><div>Our applications write to the cluster.=A0After a=
bout a day of writing we started noticing the commitlog disk filling up. So=
on we went over the disk limit and writes started failing. At this point we=
 stopped the cluster.</div>

<div>Over the course of the day we inserted around 25G of data. Our columns=
 values are pretty small.=A0</div><div>I understand that cassandra periodic=
ally cleans up the commitlog directories by generating sstables in datadir.=
 Is there any way to speed up this movement from commitog to datadir?=A0</d=
iv>

<div>Thanks!</div><div><br></div>
</blockquote></div><br></div></div></div>

--000e0cd2e71669297d04a3a3f24e--

--000e0cd2e71669299904a3a3f250
Content-Type: application/octet-stream; name="PeriodicMemtableFlusher.java"
Content-Disposition: attachment; filename="PeriodicMemtableFlusher.java"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_gnvy1wem0

cGFja2FnZSBjb20uc2ltcGxlZ2VvLnBlbmVsb3BlLmRiOwoKaW1wb3J0IG9yZy5hcGFjaGUuY2Fz
c2FuZHJhLmNvbmZpZy5EYXRhYmFzZURlc2NyaXB0b3I7CmltcG9ydCBvcmcuYXBhY2hlLmNhc3Nh
bmRyYS5kYi5Db2x1bW5GYW1pbHlTdG9yZTsKaW1wb3J0IG9yZy5hcGFjaGUuY2Fzc2FuZHJhLmRi
LlRhYmxlOwppbXBvcnQgb3JnLmFwYWNoZS5sb2c0ai5Mb2dnZXI7CgppbXBvcnQgamF2YS5pby5G
aWxlOwppbXBvcnQgamF2YS5pby5GaWxlbmFtZUZpbHRlcjsKaW1wb3J0IGphdmEuaW8uSU9FeGNl
cHRpb247CmltcG9ydCBqYXZhLnV0aWwuKjsKaW1wb3J0IGphdmEudXRpbC5jb25jdXJyZW50LlRp
bWVVbml0OwoKcHVibGljIGNsYXNzIFBlcmlvZGljTWVtdGFibGVGbHVzaGVyIHsKICAgIHByaXZh
dGUgc3RhdGljIGZpbmFsIExvZ2dlciBsb2dnZXIgPSBMb2dnZXIuZ2V0TG9nZ2VyKFBlcmlvZGlj
TWVtdGFibGVGbHVzaGVyLmNsYXNzKTsKCiAgICBwcml2YXRlIGZpbmFsIGxvbmcgdGhyZXNob2xk
OwogICAgcHJpdmF0ZSBmaW5hbCBUaW1lciB0aW1lcjsKCiAgICAvKioKICAgICAqIENoZWNrcyB0
aGUgdG90YWwgc2l6ZSBvZiBjb21taXQgbG9ncyBwZXJpb2RpY2FsbHkgYW5kLCBpZiB0aGV5J3Jl
IGdyZWF0ZXIKICAgICAqIHRoYW4gc2l6ZSwgZmx1c2hlcyBtZW10YWJsZXMgc28gdGhhdCB0aGUg
Y29tbWl0IGxvZ3MgY2FuIGJlIHJlbW92ZWQuCiAgICAgKgogICAgICogQHBhcmFtIGludGVydmFs
IGhvdyBvZnRlbiB3ZSBzaG91bGQgY2hlY2sgY29tbWl0IGxvZyBzaXplCiAgICAgKiBAcGFyYW0g
dW5pdCB0aGUgdW5pdCBvZiBpbnRlcnZhbAogICAgICogQHBhcmFtIHRocmVzaG9sZCB0aGUgc2l6
ZSBvZiBjb21taXQgbG9ncywgaW4gYnl0ZXMsIGJlZm9yZSB3ZSBzaG91bGQgZmx1c2gKICAgICAq
LwogICAgcHVibGljIFBlcmlvZGljTWVtdGFibGVGbHVzaGVyKGxvbmcgaW50ZXJ2YWwsIFRpbWVV
bml0IHVuaXQsIGxvbmcgdGhyZXNob2xkKSB7CiAgICAgICAgLy8gbWlnaHQgd2FudCB0byBhZGQg
YSBzYWZlZ3VhcmQgdGhhdCB3b24ndCBsZXQgeW91IHNldCB0aHJlc2hvbGQgPCBjb21taXQgbG9n
IHNlZ21lbnQgc2l6ZQogICAgICAgIHRoaXMudGhyZXNob2xkID0gdGhyZXNob2xkOwogICAgICAg
IHRoaXMudGltZXIgPSBuZXcgVGltZXIoKTsKICAgICAgICB0aGlzLnRpbWVyLnNjaGVkdWxlKG5l
dyBUaW1lclRhc2soKSB7CiAgICAgICAgICAgIHB1YmxpYyB2b2lkIHJ1bigpIHsKICAgICAgICAg
ICAgICAgIFBlcmlvZGljTWVtdGFibGVGbHVzaGVyLnRoaXMubWF5YmVGbHVzaFNTVGFibGVzKCk7
CiAgICAgICAgICAgIH0KICAgICAgICB9LCBUaW1lVW5pdC5NSUxMSVNFQ09ORFMuY29udmVydChp
bnRlcnZhbCwgdW5pdCksIFRpbWVVbml0Lk1JTExJU0VDT05EUy5jb252ZXJ0KGludGVydmFsLCB1
bml0KSk7CiAgICB9CgogICAgcHVibGljIGJvb2xlYW4gbWF5YmVGbHVzaFNTVGFibGVzKCkgewog
ICAgICAgIGxvbmcgc2l6ZSA9IGNvbW1pdExvZ1NpemUoKTsKICAgICAgICBpZiAoc2l6ZSA+PSB0
aHJlc2hvbGQpIHsKICAgICAgICAgICAgbG9nZ2VyLmluZm8oIkNvbW1pdCBsb2cgc2l6ZTogIiAr
IHNpemUgKyAiIGlzIGFib3ZlIGZsdXNoIHRocmVzaG9sZDogIiArIHRocmVzaG9sZCArICIgZmx1
c2hpbmcgbWVtdGFibGVzLiIpOwoKICAgICAgICAgICAgZm9yIChTdHJpbmcgbmFtZSA6IHRhYmxl
cygpKSB7CiAgICAgICAgICAgICAgICB0cnkgewogICAgICAgICAgICAgICAgICAgIFRhYmxlIHRh
YmxlID0gVGFibGUub3BlbihuYW1lKTsKICAgICAgICAgICAgICAgICAgICBmb3IgKENvbHVtbkZh
bWlseVN0b3JlIHN0b3JlIDogdGFibGUuZ2V0Q29sdW1uRmFtaWx5U3RvcmVzKCkpIHsKICAgICAg
ICAgICAgICAgICAgICAgICAgc3RvcmUuZm9yY2VGbHVzaCgpOwogICAgICAgICAgICAgICAgICAg
IH0KICAgICAgICAgICAgICAgIH0gY2F0Y2ggKElPRXhjZXB0aW9uIGUpIHsKICAgICAgICAgICAg
ICAgICAgICBsb2dnZXIuZXJyb3IoIkVycm9yIGZsdXNoaW5nIFNTVGFibGU6ICIgKyBuYW1lLCBl
KTsKICAgICAgICAgICAgICAgIH0KICAgICAgICAgICAgfQogICAgICAgICAgICByZXR1cm4gdHJ1
ZTsKICAgICAgICB9IGVsc2UgewogICAgICAgICAgICBsb2dnZXIuaW5mbygiQ29tbWl0IGxvZyBz
aXplOiAiICsgc2l6ZSArICIgaXMgYmVsb3cgZmx1c2ggdGhyZXNob2xkOiAiICsgdGhyZXNob2xk
ICsgIiwgbm90aGluZyB0byBzZWUgaGVyZS4iKTsKICAgICAgICB9CiAgICAgICAgcmV0dXJuIGZh
bHNlOwogICAgfQoKICAgIHB1YmxpYyBTZXQ8U3RyaW5nPiB0YWJsZXMoKSB7CiAgICAgICAgcmV0
dXJuIERhdGFiYXNlRGVzY3JpcHRvci5nZXRUYWJsZXMoKTsKICAgIH0KCiAgICBwdWJsaWMgbG9u
ZyBjb21taXRMb2dTaXplKCkgewogICAgICAgIFN0cmluZyBkaXJlY3RvcnkgPSBEYXRhYmFzZURl
c2NyaXB0b3IuZ2V0TG9nRmlsZUxvY2F0aW9uKCk7CiAgICAgICAgTGlzdDxGaWxlPiBmaWxlcyA9
IEFycmF5cy5hc0xpc3QobmV3IEZpbGUoZGlyZWN0b3J5KS5saXN0RmlsZXMobmV3IEZpbGVuYW1l
RmlsdGVyKCkgewogICAgICAgICAgICBwdWJsaWMgYm9vbGVhbiBhY2NlcHQoRmlsZSBkaXIsIFN0
cmluZyBuYW1lKSB7CiAgICAgICAgICAgICAgICByZXR1cm4gbmFtZS5tYXRjaGVzKCJDb21taXRM
b2ctXFxkKy5sb2ciKTsKICAgICAgICAgICAgfQogICAgICAgIH0pKTsKCiAgICAgICAgbG9uZyBz
aXplID0gMDsKICAgICAgICBmb3IgKEZpbGUgZmlsZSA6IGZpbGVzKSB7CiAgICAgICAgICAgIHNp
emUgKz0gZmlsZS5sZW5ndGgoKTsKICAgICAgICB9CiAgICAgICAgcmV0dXJuIHNpemU7CiAgICB9
Cn0K
--000e0cd2e71669299904a3a3f250--