incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suhail Doshi <suh...@mixpanel.com>
Subject Re: Frequent timeout exceptions on Cassandra
Date Mon, 01 Feb 2010 17:32:57 GMT
One last thing, timeouts start happening to happen more frequently either
with time or just more reads/writes.

Suhail

On Mon, Feb 1, 2010 at 9:31 AM, Suhail Doshi <suhail@mixpanel.com> wrote:

> So it turns out it's not socket related and it seems to only happen on my
> second new node only not the first one with a higher load.
>
> I decommissioned the new node and replaced it with another new node from
> scratch but the issue persists: clients continue to timeout on it and
> eventual the node becomes unavailable after a while requiring a restart.
>
> I've gone as far as I can trying to debug it so would greatly appreciate
> any insight. Let me know if there's anything else you need to know.
>
> Suhail
>
>  On Sun, Jan 31, 2010 at 2:55 PM, Suhail Doshi <suhail@mixpanel.com>wrote:
>
>> Encountering some new issues while running clients against Cassandra. For some reason
as time goes Cassandra in my case starts to throw timeout exceptions to the point where even
if one client is connected it processes things *very* slow and often throws the timeout exception
frequently. My clients do: Write ops: ~2.5-4.5k, Read ops: ~7-10ku
>>
>> Does anyone have a clue what might cause the timeout errors? Things seem to get progressively
worse as time goes on.
>>
>> I wrote my own client using Thrift's python output of the thrift interface. Pretty
standard and close to the example. open transport -> execute query -> close transport.
I am wondering if possibly I shouldn't close transport on every query and leave it open. Potentially
opening and closing the socket to the thrift server could cause problems at scale?
>>
>> I checked the usual suspects: iostat, vmstat -- all is fine. Plenty of memory on
the 2 64-bit 8 G machines.
>>
>>
>>
>>
>> First run
>>
>> ----------------------
>> total,interval_op_rate,avg_latency,elapsed_time28727,2872,0.0175149779964,1061622,3289,0.0146141170531,20101748,4012,0.0119166439863,30137757,3600,0.0135236120405,40175646,3788,0.012693485803,50213738,3809,0.0126799201372,60245012,3127,0.0155579636391,70285269,4025,0.0118225181672,80324749,3948,0.0121112306852,90355683,3093,0.0157108579878,100388473,3279,0.0148143836004,110426829,3835,0.0125468635425,120463487,3665,0.0133001198517,130502343,3885,0.0124575762634,140533697,3135,0.0154061204143,150567714,3401,0.0135120152761,160603673,3595,0.0141025309853,170643477,3980,0.0120242987251,180683208,3973,0.0120188054736,190723563,4035,0.0118774686127,200753554,2999,0.0160202514827,210787872,3431,0.0141030247447,220828373,4050,0.0118379161928,230868246,3987,0.012024458323,240907380,3913,0.0122560267615,250937451,3007,0.01239524122,260975023,3757,0.0156890653541,2701000000,2497,0.0130479447468,280
>> Second run (ho later)
>>
>> ----------------------
>> total,interval_op_rate,avg_latency,elapsed_time25631,2563,0.0192476335427,1059575,3394,0.0139286767445,2095670,3609,0.0132010291332,30111244,1557,0.0317069589671,40131045,1980,0.0245688458379,50141317,1027,0.0364191305368,60150975,965,0.0632473327625,70183505,3253,0.0147324700155,80215452,3194,0.0148266515631,90252952,3750,0.0128544898542,100286385,3343,0.0143946225061,110322792,3640,0.0128910038463,120338928,1613,0.0288048155041,130369871,3094,0.0167357261351,140398297,2842,0.017077029762,150415342,1704,0.0288302706917,160429611,1426,0.0346701375921,170461401,3179,0.0147932940138,180Process
Inserter-22:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()478302,1690,0.0294986360419,190511967,3366,0.014146656114,200544990,3302,0.0143403996201,210579351,3436,0.0138775690627,220612898,3354,0.0141493124013,230647257,3435,0.0137730139321,240669652,2239,0.0131938768354,250685008,1535,0.0435601793934,260695135,1012,0.0371799632852,270719213,2407,0.0244178672953,280753296,3408,0.0139683472405,290786566,3327,0.0142738517283,300812512,2594,0.0146942751213,310831476,1896,0.0302928054768,320862522,3104,0.0150721740732,330893463,3094,0.0155329019027,340910865,1740,0.0276735682636,350915065,420,0.0715707530862,360927336,1227,0.0543709511221,370944975,1763,0.0258861654665,380963617,1864,0.0244650796636,390981487,1787,0.028689940935,400989352,786,0.00684437982155,410
>> Third run (3 hours later)Note: According to jconsole write operations were < 10,000
but during the first run I saw them peak at 80,000. Huge difference.----------------------
>> total,interval_op_rate,avg_latency,elapsed_timeProcess Inserter-26:Traceback (most
recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-4:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
>>     raise result.ueUnavailableException: UnavailableException()Process Inserter-6:Traceback
(most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
>>     raise result.ueUnavailableException: UnavailableException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
>>     raise result.ueUnavailableException: UnavailableException()9892,989,0.0452791625702,1013884,399,0.0905211865902,20Process
Inserter-9:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()16919,303,0.113362641861,3017398,47,1.34807682635,4017735,33,0.934142259179,5018103,36,1.70099910122,6018596,49,0.666715461874,7018867,27,2.30892864716,80Process
Inserter-45:Traceback (most recent call last):Process Inserter-23:Traceback (most recent call
last):Process Inserter-29:Process Inserter-32:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.run()
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-20:Traceback (most recent call last):Process Inserter-42:Process
Inserter-33:Traceback (most recent call last):Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTraceback (most recent call last):TimedOutException: TimedOutException()
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in runProcess Inserter-34:Traceback (most recent call
last):Process Inserter-8:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-24:Traceback (most recent call last):
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-47:Traceback (most recent call last):
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()Process Inserter-36:Traceback
(most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-1:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-2:Traceback (most recent call last):
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     raise result.teTimedOutException: TimedOutException()
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()Process Inserter-37:Traceback
(most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.run()
>>   File "stress.py", line 167, in run
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insertProcess Inserter-38:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     raise result.teTimedOutException: TimedOutException()
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-7:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()19660,79,0.259248053742,9026066,640,0.0368563048809,10034222,815,0.0542649867606,11037799,357,0.0611899578275,12042595,479,0.047750012292,13042814,21,1.82856085638,14042863,4,4.05655763101,150Process
Inserter-18:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in runProcess Inserter-39:Traceback (most recent call
last):
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()Process Inserter-12:Traceback
(most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-19:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()Process Inserter-35:Traceback
(most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-41:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()Process Inserter-15:
>>     self.run()Traceback (most recent call last):
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()43380,51,0.592962222588,16043556,17,0.85316223448,17043582,2,5.76186679877,18044110,52,0.574297892776,19044143,3,4.52656103625,20044799,65,0.46613632397,210
>>
>>
>> 44828,2,5.20553808377,22044976,14,2.0302237675,230Process Inserter-31:Traceback (most
recent call last):Process Inserter-11:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-30:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-27:Traceback (most recent call last):
>>     self.run()
>>   File "stress.py", line 167, in run
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()Process Inserter-46:Traceback
(most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()45324,34,0.361710271616,240Process
Inserter-44:Traceback (most recent call last):Process Inserter-21:Traceback (most recent call
last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-22:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()Process Inserter-49:Traceback
(most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()45394,7,2.14389359951,250Process
Inserter-10:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()45413,1,6.80838800731,26045836,42,0.212721344427,270Process
Inserter-13:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()45872,3,4.39313924313,28045899,2,2.80208562039,29046025,12,1.03094763226,30046036,1,6.91312122345,310Process
Inserter-5:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()46041,0,9.6405497551,32046937,89,0.210266448291,33047675,73,0.0936150796368,34049555,188,0.0744233962069,35050292,73,0.0938777962404,36051306,101,0.136950609717,37052649,134,0.0525897449286,38053247,59,0.229411067771,39053888,64,0.107282243541,40054971,108,0.0673501024836,410Process
Inserter-28:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-3:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-16:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.te
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insertTimedOutException: TimedOutException()
>>     raise result.teTimedOutException: TimedOutException()55637,66,0.174130035234,42055875,23,0.201759393476,43057258,138,0.0706880513258,44057522,26,0.182813448436,450Process
Inserter-25:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in runProcess Inserter-17:Traceback (most recent call
last):
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()58338,81,0.102476425615,46058565,22,0.151250916956,47058695,13,0.266876726884,48059481,78,0.0896081578641,49059767,28,0.121027883116,50061782,201,0.0350987749123,51066830,504,0.00852425800262,52067753,92,0.0383770502014,53067789,3,1.9088866512,54067789,0,NAN,550Process
Inserter-50:Process Inserter-40:Traceback (most recent call last):Traceback (most recent call
last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in runProcess Inserter-14:
>>     self.run()
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "stress.py", line 167, in run
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertTraceback (most recent call last):
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()67902,11,0.370109583424,560Process
Inserter-48:Traceback (most recent call last):
>>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>>     self.run()
>>   File "stress.py", line 167, in run
>>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
>>     self.recv_batch_insert()
>>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
>>     raise result.teTimedOutException: TimedOutException()67903,0,6.85239100456,57068002,9,0.13895741376,58068003,0,6.84438991547,590
>>
>> Extra information:
>>
>> 1st no clients were connected when it was run
>> 2nd run, stress test was conducted when there were 50+ clients doing 7k read, 3-4k
write operations
>> 3rd was run after clients were killed as problems showed up
>> After 3rd run looked at tpstats for seed-node:Pool Name                    Active
  Pending      CompletedFILEUTILS-DELETE-POOL             0         0            442STREAM-STAGE
                     0         0              0RESPONSE-STAGE                    0       
 0        1427177ROW-READ-STAGE                    0         0        3705481LB-OPERATIONS
                    0         0              0COMMITLOG                         1        
0        5133614MESSAGE-DESERIALIZER-POOL         0         0        3456220GMFD         
                    0         0          34882LB-TARGET                         0        
0              0CONSISTENCY-MANAGER               0         0              0ROW-MUTATION-STAGE
               0         0        5452268MESSAGE-STREAMING-POOL            0         0   
          0LOAD-BALANCER-STAGE               0         0              0FLUSH-SORTER-POOL 
               0         0            118MEMTABLE-POST-FLUSHER             0         0   
        118FLUSH-WRITER-POOL                 0         0            118AE-SERVICE-STAGE  
               0         0              1HINTED-HANDOFF-POOL               0         0   
         41
>>
>>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>



-- 
http://mixpanel.com
Blog: http://blog.mixpanel.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message