incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Stress tests failed with secondary index
Date Thu, 07 Apr 2011 01:27:32 GMT
TimedOutException means that the less than CL number of nodes responded to the coordinator
before the rpc_timeout.

So it was overloaded. Which makes sense when you say it only happens with secondary indexes.
Consider things like
- reducing the throughput
- reducing the number of clients
- ensuring the clients are connecting to all nodes in the cluster. 

You will probably find some logs about dropped messages on some nodes.
Aaron
 
On 6 Apr 2011, at 20:39, Sheng Chen wrote:

> I used py_stress module to insert 10m test data with a secondary index.
> I got the following exceptions.
> 
> # python stress.py -d xxx -o insert -n 10000000 -c 5 -s 34 -C 5 -x keys
> total,interval_op_rate,interval_key_rate,avg_latency,elapsed_time
> 265322,26532,26541,0.00186140829433,10
> 630300,36497,36502,0.00129331431204,20
> 986781,35648,35640,0.0013310986218,30
> 1332190,34540,34534,0.00135942295893,40
> 1473578,14138,14138,0.00142941070007,50
> Process Inserter-38:
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/multiprocessing/process.py", line 237, in
_bootstrap
>     self.run()
>   File "stress.py", line 242, in run
>     self.cclient.batch_mutate(cfmap, consistency)
>   File "/root/apache-cassandra-0.7.4-src/interface/thrift/gen-py/cassandra/Cassandra.py",
line 784, in batch_mutate
> TimedOutException: TimedOutException(args=())
>     self.run()
>   File "stress.py", line 242, in run
>     self.recv_batch_mutate()
>   File "/root/apache-cassandra-0.7.4-src/interface/thrift/gen-py/cassandra/Cassandra.py",
line 810, in recv_batch_mutate
>     raise result.te
> 
> 
> Tests without secondary index is ok at about 40k ops/sec.
> 
> There is a `GC for ParNew` for about 200ms taking place every second. Does it matter?
> The same gc for about 400ms happens every 2 seconds, which does not hurt the inserts
without secondary index.
> 
> Thanks in advance for any advice.
> 
> Sheng


Mime
View raw message