cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suhail Doshi <suh...@mixpanel.com>
Subject Frequent timeout exceptions on Cassandra
Date Sun, 31 Jan 2010 22:55:33 GMT
Encountering some new issues while running clients against Cassandra.
For some reason as time goes Cassandra in my case starts to throw
timeout exceptions to the point where even if one client is connected
it processes things *very* slow and often throws the timeout exception
frequently. My clients do: Write ops: ~2.5-4.5k, Read ops: ~7-10ku

Does anyone have a clue what might cause the timeout errors? Things
seem to get progressively worse as time goes on.

I wrote my own client using Thrift's python output of the thrift
interface. Pretty standard and close to the example. open transport ->
execute query -> close transport. I am wondering if possibly I
shouldn't close transport on every query and leave it open.
Potentially opening and closing the socket to the thrift server could
cause problems at scale?

I checked the usual suspects: iostat, vmstat -- all is fine. Plenty of
memory on the 2 64-bit 8 G machines.


First run

----------------------
total,interval_op_rate,avg_latency,elapsed_time28727,2872,0.0175149779964,1061622,3289,0.0146141170531,20101748,4012,0.0119166439863,30137757,3600,0.0135236120405,40175646,3788,0.012693485803,50213738,3809,0.0126799201372,60245012,3127,0.0155579636391,70285269,4025,0.0118225181672,80324749,3948,0.0121112306852,90355683,3093,0.0157108579878,100388473,3279,0.0148143836004,110426829,3835,0.0125468635425,120463487,3665,0.0133001198517,130502343,3885,0.0124575762634,140533697,3135,0.0154061204143,150567714,3401,0.0135120152761,160603673,3595,0.0141025309853,170643477,3980,0.0120242987251,180683208,3973,0.0120188054736,190723563,4035,0.0118774686127,200753554,2999,0.0160202514827,210787872,3431,0.0141030247447,220828373,4050,0.0118379161928,230868246,3987,0.012024458323,240907380,3913,0.0122560267615,250937451,3007,0.01239524122,260975023,3757,0.0156890653541,2701000000,2497,0.0130479447468,280
Second run (ho later)

----------------------
total,interval_op_rate,avg_latency,elapsed_time25631,2563,0.0192476335427,1059575,3394,0.0139286767445,2095670,3609,0.0132010291332,30111244,1557,0.0317069589671,40131045,1980,0.0245688458379,50141317,1027,0.0364191305368,60150975,965,0.0632473327625,70183505,3253,0.0147324700155,80215452,3194,0.0148266515631,90252952,3750,0.0128544898542,100286385,3343,0.0143946225061,110322792,3640,0.0128910038463,120338928,1613,0.0288048155041,130369871,3094,0.0167357261351,140398297,2842,0.017077029762,150415342,1704,0.0288302706917,160429611,1426,0.0346701375921,170461401,3179,0.0147932940138,180Process
Inserter-22:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()478302,1690,0.0294986360419,190511967,3366,0.014146656114,200544990,3302,0.0143403996201,210579351,3436,0.0138775690627,220612898,3354,0.0141493124013,230647257,3435,0.0137730139321,240669652,2239,0.0131938768354,250685008,1535,0.0435601793934,260695135,1012,0.0371799632852,270719213,2407,0.0244178672953,280753296,3408,0.0139683472405,290786566,3327,0.0142738517283,300812512,2594,0.0146942751213,310831476,1896,0.0302928054768,320862522,3104,0.0150721740732,330893463,3094,0.0155329019027,340910865,1740,0.0276735682636,350915065,420,0.0715707530862,360927336,1227,0.0543709511221,370944975,1763,0.0258861654665,380963617,1864,0.0244650796636,390981487,1787,0.028689940935,400989352,786,0.00684437982155,410
Third run (3 hours later)Note: According to jconsole write operations
were < 10,000 but during the first run I saw them peak at 80,000. Huge
difference.----------------------
total,interval_op_rate,avg_latency,elapsed_timeProcess
Inserter-26:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-4:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
    raise result.ueUnavailableException: UnavailableException()Process
Inserter-6:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
    raise result.ueUnavailableException: UnavailableException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 600, in recv_batch_insert
    raise result.ueUnavailableException:
UnavailableException()9892,989,0.0452791625702,1013884,399,0.0905211865902,20Process
Inserter-9:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()16919,303,0.113362641861,3017398,47,1.34807682635,4017735,33,0.934142259179,5018103,36,1.70099910122,6018596,49,0.666715461874,7018867,27,2.30892864716,80Process
Inserter-45:Traceback (most recent call last):Process
Inserter-23:Traceback (most recent call last):Process
Inserter-29:Process Inserter-32:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.run()
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-20:Traceback (most recent
call last):Process Inserter-42:Process Inserter-33:Traceback (most
recent call last):Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.run()
  File "stress.py", line 167, in run
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTraceback (most recent call
last):TimedOutException: TimedOutException()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in runProcess Inserter-34:Traceback
(most recent call last):Process Inserter-8:Traceback (most recent call
last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-24:Traceback (most recent
call last):
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-47:Traceback (most recent call
last):
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()Process
Inserter-36:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-1:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-2:Traceback (most recent call
last):
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    raise result.teTimedOutException: TimedOutException()
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()Process
Inserter-37:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.run()
  File "stress.py", line 167, in run
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insertProcess Inserter-38:Traceback (most
recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    raise result.teTimedOutException: TimedOutException()
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-7:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()19660,79,0.259248053742,9026066,640,0.0368563048809,10034222,815,0.0542649867606,11037799,357,0.0611899578275,12042595,479,0.047750012292,13042814,21,1.82856085638,14042863,4,4.05655763101,150Process
Inserter-18:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in runProcess Inserter-39:Traceback
(most recent call last):
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()Process
Inserter-12:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-19:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()Process
Inserter-35:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-41:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()Process Inserter-15:
    self.run()Traceback (most recent call last):
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()43380,51,0.592962222588,16043556,17,0.85316223448,17043582,2,5.76186679877,18044110,52,0.574297892776,19044143,3,4.52656103625,20044799,65,0.46613632397,210


44828,2,5.20553808377,22044976,14,2.0302237675,230Process
Inserter-31:Traceback (most recent call last):Process
Inserter-11:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-30:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-27:Traceback (most recent call
last):
    self.run()
  File "stress.py", line 167, in run
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()Process
Inserter-46:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()45324,34,0.361710271616,240Process
Inserter-44:Traceback (most recent call last):Process
Inserter-21:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-22:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()Process
Inserter-49:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()45394,7,2.14389359951,250Process
Inserter-10:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()45413,1,6.80838800731,26045836,42,0.212721344427,270Process
Inserter-13:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()45872,3,4.39313924313,28045899,2,2.80208562039,29046025,12,1.03094763226,30046036,1,6.91312122345,310Process
Inserter-5:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()46041,0,9.6405497551,32046937,89,0.210266448291,33047675,73,0.0936150796368,34049555,188,0.0744233962069,35050292,73,0.0938777962404,36051306,101,0.136950609717,37052649,134,0.0525897449286,38053247,59,0.229411067771,39053888,64,0.107282243541,40054971,108,0.0673501024836,410Process
Inserter-28:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-3:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertProcess Inserter-16:Traceback (most recent
call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.te
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insertTimedOutException: TimedOutException()
    raise result.teTimedOutException:
TimedOutException()55637,66,0.174130035234,42055875,23,0.201759393476,43057258,138,0.0706880513258,44057522,26,0.182813448436,450Process
Inserter-25:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in runProcess Inserter-17:Traceback
(most recent call last):
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()58338,81,0.102476425615,46058565,22,0.151250916956,47058695,13,0.266876726884,48059481,78,0.0896081578641,49059767,28,0.121027883116,50061782,201,0.0350987749123,51066830,504,0.00852425800262,52067753,92,0.0383770502014,53067789,3,1.9088866512,54067789,0,NAN,550Process
Inserter-50:Process Inserter-40:Traceback (most recent call
last):Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in runProcess Inserter-14:
    self.run()
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "stress.py", line 167, in run
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insertTraceback (most recent call last):
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException: TimedOutException()
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()67902,11,0.370109583424,560Process
Inserter-48:Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
    self.run()
  File "stress.py", line 167, in run
    self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 574, in batch_insert
    self.recv_batch_insert()
  File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py",
line 602, in recv_batch_insert
    raise result.teTimedOutException:
TimedOutException()67903,0,6.85239100456,57068002,9,0.13895741376,58068003,0,6.84438991547,590

Extra information:

1st no clients were connected when it was run
2nd run, stress test was conducted when there were 50+ clients doing
7k read, 3-4k write operations
3rd was run after clients were killed as problems showed up
After 3rd run looked at tpstats for seed-node:Pool Name
    Active   Pending      CompletedFILEUTILS-DELETE-POOL             0
        0            442STREAM-STAGE                      0         0
            0RESPONSE-STAGE                    0         0
1427177ROW-READ-STAGE                    0         0
3705481LB-OPERATIONS                     0         0
0COMMITLOG                         1         0
5133614MESSAGE-DESERIALIZER-POOL         0         0
3456220GMFD                              0         0
34882LB-TARGET                         0         0
0CONSISTENCY-MANAGER               0         0
0ROW-MUTATION-STAGE                0         0
5452268MESSAGE-STREAMING-POOL            0         0
0LOAD-BALANCER-STAGE               0         0
0FLUSH-SORTER-POOL                 0         0
118MEMTABLE-POST-FLUSHER             0         0
118FLUSH-WRITER-POOL                 0         0
118AE-SERVICE-STAGE                  0         0
1HINTED-HANDOFF-POOL               0         0             41

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message