cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thunder Stumpges <thunder.stump...@gmail.com>
Subject Re: Where to I start to get to the bottom of this WriteTimeout issue?
Date Mon, 17 Feb 2014 04:13:20 GMT
If you are looking for write throughput and running on a VM you could likely have IO issues
with your virtual disks.. Best practices are to put the write ahead log on a separate disk
from the data folder(s). Not sure if you have done this or what physical setup you have under
the VM but I would also examine your IO while you are doing this. Is there other load on the
system either read or write while this is happening?

-Thunder


> On Feb 16, 2014, at 8:04 PM, Erick Ramirez <erick@ramirez.com.au> wrote:
> 
> Jacob,
> 
> You are right in that increasing the timeout to 20,000ms (20 seconds) is a real concern
as it just hides an underlying issue with your environment. Without additional information,
I was suspecting that this could be due to the environment not being optimised.
> 
> These write timeouts can occur when the systems are under load or low on resources. My
questioning around memory is leading to the fact that your system(s) may possibly be under
load due to GC which points to JVM running out of memory.
> 
> Have a look at the logs as they will give you clues as to what is happening, and possibly
the cause of the issue. And keep us posted. Thanks!
> 
> Cheers,
> Erick
> 
> 
> 
>> On Mon, Feb 17, 2014 at 1:41 PM, Jacob Rhoden <jacob.rhoden@me.com> wrote:
>> Hi Erick,
>> 
>>> On 17 Feb 2014, at 1:19 pm, Erick Ramirez <erick@ramirez.com.au> wrote:
>>> Are you able to post log snippets around the time that the timeouts occur?
>>> 
>>> I have a suspicion you may be running out of heap memory and might need to tune
your environment. The INFO entries in the log should indicate this.
>> 
>> Im kicking off the load and not watching it so I don’t have a timestamp to see
where it occurred. After some mucking around I worked out that adding an extra zero to the
following parameter on both nodes makes the problem has gone away:
>> 
>> write_request_timeout_in_ms: 20000
>> 
>> Whatever that parameter exactly controls, Im pretty sure I don’t want to keep a
20s write timeout :D but it allows my bulk loads to run for the time being.
>> 
>> The nodes are running on some test VM’s  with xmx/xms set at 1Gb. So are you assuming
that bulk counter row adding/incrementing can cause memory issues? How much memory do you
need to allocate before this category of problem would disappear?
>> 
>> Thanks,
>> Jacob
> 

Mime
View raw message