cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From J. Amding <jamd...@gmail.com>
Subject Re: TimedOutException caused by
Date Tue, 24 Jul 2012 21:10:25 GMT


aaron morton <aaron <at> thelastpickle.com> writes:

> 
> The cluster is running into GC problems and this is slowing it down under the 
stress test. When it slows down one or more of the nodes is failing to perform 
the write within rpc_timeout . This causes the coordinator of the write to raise 
the TimedOutException. 
> You options are:
> 
> * allocate more memory
> * ease back on the stress test. 
> * work as a CL QUORUM so that one node failing does result in the error. 
> 
> see also http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
> 
> 
> Cheers
>  
> 
> 
> 
> 
> 
> -----------------
> Aaron Morton
> Freelance Developer
>  <at> aaronmorton
> http://www.thelastpickle.com
> 
> 
> 
> On 28/05/2012, at 12:59 PM, Jason Tang wrote:
> Hi
> My system is 4 nodes 64 bit cassandra cluster, 6G big per node,default 
configuration (which means 1/3 heap for memtable), replicate number 3, write 
all, read one.
> When I run stress load testing, I got this TimedOutException, and some 
operation failed, and all traffic hang for a while. 
> 
> And when I have 1G memory 32 bit cassandra on standalone model, I didn't find 
so frequently "Stop the world" behavior.
> 
> So I wonder what kind of operation will hang the cassandra system. 
> 
> 
> How to collect information for tuning.
> 
> From the system log and document, I guess there are three type operations:
> 1) Flush memtable when meet max size
> 
> 2) Compact SSTable (why?)
> 3) Java GC
> 
> system.log:
> 
>  INFO [main] 2012-05-25 16:12:17,054 ColumnFamilyStore.java (line 688) 
Enqueuing flush of Memtable-LocationInfo <at> 1229893321(53/66 serialized/live 
bytes, 2 ops)
>  INFO [FlushWriter:1] 2012-05-25 16:12:17,054 Memtable.java (line 239) Writing 
Memtable-LocationInfo <at> 1229893321(53/66 serialized/live bytes, 2 ops)
>  INFO [FlushWriter:1] 2012-05-25 16:12:17,166 Memtable.java (line 275) 
Completed flushing /var/proclog/raw/cassandra/data/system/LocationInfo-hb-2-
Data.db (163 bytes)
> 
> ...
> 
>  INFO [CompactionExecutor:441] 2012-05-28 08:02:55,345 CompactionTask.java 
(line 112) Compacting 
[SSTableReader(path='/var/proclog/raw/cassandra/data/myks/queue-hb-41-Data.db'), 
SSTableReader(path='/var/proclog/raw/cassandra/data/
> myks /queue-hb-32-Data.db'), 
SSTableReader(path='/var/proclog/raw/cassandra/data/
> myks /queue-hb-37-Data.db'), 
SSTableReader(path='/var/proclog/raw/cassandra/data/
> myks /queue-hb-53-Data.db')]
> ...
> 
> 
>  WARN [ScheduledTasks:1] 2012-05-28 08:02:26,619 GCInspector.java (line 146) 
Heap is 0.7993011015621736 full.  You may need to reduce memtable and/or cache 
sizes.  Cassandra will now flush up to the two largest memtables to free up 
memory.  Adjust flush_largest_memtables_at threshold in cassandra.yaml if you 
don't want Cassandra to do this automatically
>  INFO [ScheduledTasks:1] 2012-05-28 08:02:54,980 GCInspector.java (line 123) 
GC for ConcurrentMarkSweep: 728 ms for 2 collections, 3594946600 used; max is 
6274678784
>  INFO [ScheduledTasks:1] 2012-05-28 08:41:34,030 GCInspector.java (line 123) 
GC for ParNew: 1668 ms for 1 collections, 4171503448 used; max is 6274678784
>  INFO [ScheduledTasks:1] 2012-05-28 08:41:48,978 GCInspector.java (line 123) 
GC for ParNew: 1087 ms for 1 collections, 2623067496 used; max is 6274678784
>  INFO [ScheduledTasks:1] 2012-05-28 08:41:48,987 GCInspector.java (line 123) 
GC for ConcurrentMarkSweep: 3198 ms for 3 collections, 2623361280 used; max is 
6274678784
> 
> 
> 
> Timeout Exception:
> 
> Caused by: org.apache.cassandra.thrift.TimedOutException: null
>         at 
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19
495) ~[na:na]
>         at 
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:10
35) ~[na:na]
>         at 
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009) ~
[na:na]
>         at 
me.prettyprint.cassandra.service.KeyspaceServiceImpl$1.execute(KeyspaceServiceIm
pl.java:95) ~[na:na]
>         ... 64 common frames omitted
> 
> 
> BRs
> //Tang Weiqiang
> 
> 
> 
> 
> 
> 
> 

Hi, I've been running into the same type of issue, but on a single machine with 
CL ONE. Also a custom insertion stress utility. What would I need to do to 
address the timeouts? By allocate more memory do you mean increase heap size in 
the environment conf file?

Thanks,
J.






Mime
View raw message