incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erick Ramirez <er...@ramirez.com.au>
Subject Re: Where to I start to get to the bottom of this WriteTimeout issue?
Date Mon, 17 Feb 2014 04:04:56 GMT
Jacob,

You are right in that increasing the timeout to 20,000ms (20 seconds) is a
real concern as it just hides an underlying issue with your environment.
Without additional information, I was suspecting that this could be due to
the environment not being optimised.

These write timeouts can occur when the systems are under load or low on
resources. My questioning around memory is leading to the fact that your
system(s) may possibly be under load due to GC which points to JVM running
out of memory.

Have a look at the logs as they will give you clues as to what is
happening, and possibly the cause of the issue. And keep us posted. Thanks!

Cheers,
Erick



On Mon, Feb 17, 2014 at 1:41 PM, Jacob Rhoden <jacob.rhoden@me.com> wrote:

> Hi Erick,
>
> On 17 Feb 2014, at 1:19 pm, Erick Ramirez <erick@ramirez.com.au> wrote:
>
> Are you able to post log snippets around the time that the timeouts occur?
>
> I have a suspicion you may be running out of heap memory and might need to
> tune your environment. The INFO entries in the log should indicate this.
>
>
> Im kicking off the load and not watching it so I don't have a timestamp to
> see where it occurred. After some mucking around I worked out that adding
> an extra zero to the following parameter on both nodes makes the problem
> has gone away:
>
> write_request_timeout_in_ms: 20000
>
> Whatever that parameter exactly controls, Im pretty sure I don't want to
> keep a 20s write timeout :D but it allows my bulk loads to run for the time
> being.
>
> The nodes are running on some test VM's  with xmx/xms set at 1Gb. So are
> you assuming that bulk counter row adding/incrementing can cause memory
> issues? How much memory do you need to allocate before this category of
> problem would disappear?
>
> Thanks,
> Jacob
>

Mime
View raw message