Return-Path: 
 <user-return-39026-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org>
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id F12571021E
	for <apmail-cassandra-user-archive@www.apache.org>;
 Mon, 17 Feb 2014 04:13:56 +0000 (UTC)
Received: (qmail 36284 invoked by uid 500); 17 Feb 2014 04:13:54 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 35736 invoked by uid 500); 17 Feb 2014 04:13:52 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@cassandra.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@cassandra.apache.org>
List-Post: <mailto:user@cassandra.apache.org>
List-Id: <user.cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 35728 invoked by uid 99); 17 Feb 2014 04:13:52 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Feb 2014 04:13:51 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,MIME_QP_LONG_LINE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of thunder.stumpges@gmail.com
 designates 209.85.220.42 as permitted sender)
Received: from [209.85.220.42] (HELO mail-pa0-f42.google.com) (209.85.220.42)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Feb 2014 04:13:45 +0000
Received: by mail-pa0-f42.google.com with SMTP id kl14so14878178pab.1
        for <user@cassandra.apache.org>; Sun, 16 Feb 2014 20:13:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=references:in-reply-to:mime-version:content-transfer-encoding
         :content-type:message-id:cc:from:subject:date:to;
        bh=wDkkKb+rSPld65l4cc8llX0uovpQe9BkjSvhrbZkf04=;
        b=bC7EaI7jyyyGJkKT5r193jNPzg4KHrSHJixg+saaHy/pytScFSPojbBugWaeH4eARP
         XwzikXVLUFCr/f4dcivlH5N7H0DirosrjEDgCdo5sAvacRpT9PqasGXqhn50RhXqXKL+
         usDWZYL2FqObEh8Gfi2TnTBxETJexEBJ23sDyILSVS5qo0BBl9avY6XpjU4Vsi3xx8jY
         TzNBHml6Veir7iKp0vmky3Ka5G0BHikskxMyJgrL9QKEnGp/Pd78h36bKtt//WBB0F8p
         cRTUwPaUvnw2SCs8OLdBzzDgWrEUhgzY7W05LzYrb0ycCv2abAaOifH6trqM8f2RE93p
         Tr7g==
X-Received: by 10.68.204.231 with SMTP id lb7mr10006305pbc.30.1392610404071;
        Sun, 16 Feb 2014 20:13:24 -0800 (PST)
Received: from [192.168.37.145] (ip72-199-220-200.sd.sd.cox.net.
 [72.199.220.200])
        by mx.google.com with ESMTPSA id
 iq10sm40936059pbc.14.2014.02.16.20.13.22
        for <multiple recipients>
        (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
        Sun, 16 Feb 2014 20:13:22 -0800 (PST)
References: <D8281011-ADA2-452F-8743-C82576CD2F73@me.com>
 <CAB=jy0ivHTb6CahjMaK0QVkih7ad7Rgidd6Mt+_x0g6tmbANYA@mail.gmail.com>
 <10345B9E-DA6E-424B-8242-40F8DE24B78E@me.com>
 <CAB=jy0jywz0R5Ey6SNX3vp7JU7dFn0pWK1=hhjg4eeZ+n=2XDQ@mail.gmail.com>
In-Reply-To: 
 <CAB=jy0jywz0R5Ey6SNX3vp7JU7dFn0pWK1=hhjg4eeZ+n=2XDQ@mail.gmail.com>
Mime-Version: 1.0 (1.0)
Content-Transfer-Encoding: 7bit
Content-Type: multipart/alternative;
	boundary=Apple-Mail-15460A3F-CB57-42CF-B92B-684368846386
Message-Id: <BAD049BB-92FE-42EF-B3A1-F7E0AC24E995@gmail.com>
Cc: "user@cassandra.apache.org" <user@cassandra.apache.org>
X-Mailer: iPhone Mail (11B554a)
From: Thunder Stumpges <thunder.stumpges@gmail.com>
Subject: Re: Where to I start to get to the bottom of this WriteTimeout issue?
Date: Sun, 16 Feb 2014 20:13:20 -0800
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
X-Virus-Checked: Checked by ClamAV on apache.org


--Apple-Mail-15460A3F-CB57-42CF-B92B-684368846386
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

If you are looking for write throughput and running on a VM you could likely=
 have IO issues with your virtual disks.. Best practices are to put the writ=
e ahead log on a separate disk from the data folder(s). Not sure if you have=
 done this or what physical setup you have under the VM but I would also exa=
mine your IO while you are doing this. Is there other load on the system eit=
her read or write while this is happening?

-Thunder


> On Feb 16, 2014, at 8:04 PM, Erick Ramirez <erick@ramirez.com.au> wrote:
>=20
> Jacob,
>=20
> You are right in that increasing the timeout to 20,000ms (20 seconds) is a=
 real concern as it just hides an underlying issue with your environment. Wi=
thout additional information, I was suspecting that this could be due to the=
 environment not being optimised.
>=20
> These write timeouts can occur when the systems are under load or low on r=
esources. My questioning around memory is leading to the fact that your syst=
em(s) may possibly be under load due to GC which points to JVM running out o=
f memory.
>=20
> Have a look at the logs as they will give you clues as to what is happenin=
g, and possibly the cause of the issue. And keep us posted. Thanks!
>=20
> Cheers,
> Erick
>=20
>=20
>=20
>> On Mon, Feb 17, 2014 at 1:41 PM, Jacob Rhoden <jacob.rhoden@me.com> wrote=
:
>> Hi Erick,
>>=20
>>> On 17 Feb 2014, at 1:19 pm, Erick Ramirez <erick@ramirez.com.au> wrote:
>>> Are you able to post log snippets around the time that the timeouts occu=
r?
>>>=20
>>> I have a suspicion you may be running out of heap memory and might need t=
o tune your environment. The INFO entries in the log should indicate this.
>>=20
>> Im kicking off the load and not watching it so I don=E2=80=99t have a tim=
estamp to see where it occurred. After some mucking around I worked out that=
 adding an extra zero to the following parameter on both nodes makes the pro=
blem has gone away:
>>=20
>> write_request_timeout_in_ms: 20000
>>=20
>> Whatever that parameter exactly controls, Im pretty sure I don=E2=80=99t w=
ant to keep a 20s write timeout :D but it allows my bulk loads to run for th=
e time being.
>>=20
>> The nodes are running on some test VM=E2=80=99s  with xmx/xms set at 1Gb.=
 So are you assuming that bulk counter row adding/incrementing can cause mem=
ory issues? How much memory do you need to allocate before this category of p=
roblem would disappear?
>>=20
>> Thanks,
>> Jacob
>=20

--Apple-Mail-15460A3F-CB57-42CF-B92B-684368846386
Content-Type: text/html;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"content-type" content=3D"text/html; charset=3D=
utf-8"></head><body dir=3D"auto"><div>If you are looking for write throughpu=
t and running on a VM you could likely have IO issues with your virtual disk=
s.. Best practices are to put the write ahead log on a separate disk from th=
e data folder(s). Not sure if you have done this or what physical setup you h=
ave under the VM but I would also examine your IO while you are doing this. I=
s there other load on the system either read or write while this is happenin=
g?</div><div><br></div><div>-Thunder</div><div><br></div><div><br>On Feb 16,=
 2014, at 8:04 PM, Erick Ramirez &lt;<a href=3D"mailto:erick@ramirez.com.au"=
>erick@ramirez.com.au</a>&gt; wrote:<br><br></div><blockquote type=3D"cite">=
<div><div dir=3D"ltr">Jacob,<div><br></div><div>You are right in that increa=
sing the timeout to 20,000ms (20 seconds) is a real concern as it just hides=
 an underlying issue with your environment. Without additional information, I=
 was suspecting that this could be due to the environment not being optimise=
d.</div>
<div><br></div><div>These write timeouts can occur when the systems are unde=
r load or low on resources. My questioning around memory is leading to the f=
act that your system(s) may possibly be under load due to GC which points to=
 JVM running out of memory.</div>
<div><br></div><div>Have a look at the logs as they will give you clues as t=
o what is happening, and possibly the cause of the issue. And keep us posted=
. Thanks!<br><div class=3D"gmail_extra"><br clear=3D"all"><div>Cheers,<br>
Erick<br><br></div>
<br><br><div class=3D"gmail_quote">On Mon, Feb 17, 2014 at 1:41 PM, Jacob Rh=
oden <span dir=3D"ltr">&lt;<a href=3D"mailto:jacob.rhoden@me.com" target=3D"=
_blank">jacob.rhoden@me.com</a>&gt;</span> wrote:<br><blockquote class=3D"gm=
ail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-lef=
t:1ex">
<div style=3D"word-wrap:break-word">Hi Erick,<div><br></div><div><div><div>O=
n 17 Feb 2014, at 1:19 pm, Erick Ramirez &lt;<a href=3D"mailto:erick@ramirez=
.com.au" target=3D"_blank">erick@ramirez.com.au</a>&gt; wrote:</div><blockqu=
ote type=3D"cite">
<div dir=3D"ltr"><div>Are you able to post log snippets around the time that=
 the timeouts occur?</div><div><br></div><div>I have a suspicion you may be r=
unning out of heap memory and might need to tune your environment. The INFO e=
ntries in the log should indicate this.<br>

</div></div></blockquote></div><br></div><div>Im kicking off the load and no=
t watching it so I don=E2=80=99t have a timestamp to see where it occurred. A=
fter some mucking around I worked out that adding an extra zero to the follo=
wing parameter on both nodes makes the problem has gone away:</div>
<div><br></div><div><div style=3D"margin:0px;font-size:11px;font-family:Menl=
o">write_request_timeout_in_ms: 20000</div></div><div><br></div><div>Whateve=
r that parameter exactly controls, Im pretty sure I don=E2=80=99t want to ke=
ep a 20s write timeout :D but it allows my bulk loads to run for the time be=
ing.</div>
<div><br></div><div>The nodes are running on some test VM=E2=80=99s &nbsp;wi=
th xmx/xms set at 1Gb. So are you assuming that bulk counter row adding/incr=
ementing can cause memory issues? How much memory do you need to allocate be=
fore this category of problem would disappear?</div>
<div><br></div><div>Thanks,</div><div>Jacob</div></div></blockquote></div><b=
r></div></div></div>
</div></blockquote></body></html>=

--Apple-Mail-15460A3F-CB57-42CF-B92B-684368846386--