incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suhail Doshi <suh...@mixpanel.com>
Subject Re: Frequent timeout exceptions on Cassandra
Date Mon, 01 Feb 2010 17:31:38 GMT
So it turns out it's not socket related and it seems to only happen on my
second new node only not the first one with a higher load.

I decommissioned the new node and replaced it with another new node from
scratch but the issue persists: clients continue to timeout on it and
eventual the node becomes unavailable after a while requiring a restart.

I've gone as far as I can trying to debug it so would greatly appreciate any
insight. Let me know if there's anything else you need to know.

Suhail

On Sun, Jan 31, 2010 at 2:55 PM, Suhail Doshi <suhail@mixpanel.com> wrote:

> Encountering some new issues while running clients against Cassandra. For some reason
as time goes Cassandra in my case starts to throw timeout exceptions to the point where even
if one client is connected it processes things *very* slow and often throws the timeout exception
frequently. My clients do: Write ops: ~2.5-4.5k, Read ops: ~7-10ku
>
> Does anyone have a clue what might cause the timeout errors? Things seem to get progressively
worse as time goes on.
>
> I wrote my own client using Thrift's python output of the thrift interface. Pretty standard
and close to the example. open transport -> execute query -> close transport. I am wondering
if possibly I shouldn't close transport on every query and leave it open. Potentially opening
and closing the socket to the thrift server could cause problems at scale?
>
> I checked the usual suspects: iostat, vmstat -- all is fine. Plenty of memory on the
2 64-bit 8 G machines.
>
>
>
> First run
>
> ----------------------
> total,interval_op_rate,avg_latency,elapsed_time28727,2872,0.0175149779964,1061622,3289,0.0146141170531,20101748,4012,0.0119166439863,30137757,3600,0.0135236120405,40175646,3788,0.012693485803,50213738,3809,0.0126799201372,60245012,3127,0.0155579636391,70285269,4025,0.0118225181672,80324749,3948,0.0121112306852,90355683,3093,0.0157108579878,100388473,3279,0.0148143836004,110426829,3835,0.0125468635425,120463487,3665,0.0133001198517,130502343,3885,0.0124575762634,140533697,3135,0.0154061204143,150567714,3401,0.0135120152761,160603673,3595,0.0141025309853,170643477,3980,0.0120242987251,180683208,3973,0.0120188054736,190723563,4035,0.0118774686127,200753554,2999,0.0160202514827,210787872,3431,0.0141030247447,220828373,4050,0.0118379161928,230868246,3987,0.012024458323,240907380,3913,0.0122560267615,250937451,3007,0.01239524122,260975023,3757,0.0156890653541,2701000000,2497,0.0130479447468,280
> Second run (ho later)
>
> ----------------------
> total,interval_op_rate,avg_latency,elapsed_time25631,2563,0.0192476335427,1059575,3394,0.0139286767445,2095670,3609,0.0132010291332,30111244,1557,0.0317069589671,40131045,1980,0.0245688458379,50141317,1027,0.0364191305368,60150975,965,0.0632473327625,70183505,3253,0.0147324700155,80215452,3194,0.0148266515631,90252952,3750,0.0128544898542,100286385,3343,0.0143946225061,110322792,3640,0.0128910038463,120338928,1613,0.0288048155041,130369871,3094,0.0167357261351,140398297,2842,0.017077029762,150415342,1704,0.0288302706917,160429611,1426,0.0346701375921,170461401,3179,0.0147932940138,180Process
Inserter-22:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()478302,1690,0.0294986360419,190511967,3366,0.014146656114,200544990,3302,0.0143403996201,210579351,3436,0.0138775690627,220612898,3354,0.0141493124013,230647257,3435,0.0137730139321,240669652,2239,0.0131938768354,250685008,1535,0.0435601793934,260695135,1012,0.0371799632852,270719213,2407,0.0244178672953,280753296,3408,0.0139683472405,290786566,3327,0.0142738517283,300812512,2594,0.0146942751213,310831476,1896,0.0302928054768,320862522,3104,0.0150721740732,330893463,3094,0.0155329019027,340910865,1740,0.0276735682636,350915065,420,0.0715707530862,360927336,1227,0.0543709511221,370944975,1763,0.0258861654665,380963617,1864,0.0244650796636,390981487,1787,0.028689940935,400989352,786,0.00684437982155,410
> Third run (3 hours later)Note: According to jconsole write operations were < 10,000
but during the first run I saw them peak at 80,000. Huge difference.----------------------
> total,interval_op_rate,avg_latency,elapsed_timeProcess Inserter-26:Traceback (most recent
call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-4:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
600, in recv_batch_insert
>     raise result.ueUnavailableException: UnavailableException()Process Inserter-6:Traceback
(most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
600, in recv_batch_insert
>     raise result.ueUnavailableException: UnavailableException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
600, in recv_batch_insert
>     raise result.ueUnavailableException: UnavailableException()9892,989,0.0452791625702,1013884,399,0.0905211865902,20Process
Inserter-9:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()16919,303,0.113362641861,3017398,47,1.34807682635,4017735,33,0.934142259179,5018103,36,1.70099910122,6018596,49,0.666715461874,7018867,27,2.30892864716,80Process
Inserter-45:Traceback (most recent call last):Process Inserter-23:Traceback (most recent call
last):Process Inserter-29:Process Inserter-32:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.run()
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-20:Traceback (most recent call last):Process Inserter-42:Process
Inserter-33:Traceback (most recent call last):Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.run()
>   File "stress.py", line 167, in run
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTraceback (most recent call last):TimedOutException: TimedOutException()
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in runProcess Inserter-34:Traceback (most recent call last):Process
Inserter-8:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-24:Traceback (most recent call last):
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-47:Traceback (most recent call last):
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()Process Inserter-36:Traceback
(most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-1:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-2:Traceback (most recent call last):
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     raise result.teTimedOutException: TimedOutException()
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()Process Inserter-37:Traceback
(most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.run()
>   File "stress.py", line 167, in run
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insertProcess Inserter-38:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     raise result.teTimedOutException: TimedOutException()
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-7:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()19660,79,0.259248053742,9026066,640,0.0368563048809,10034222,815,0.0542649867606,11037799,357,0.0611899578275,12042595,479,0.047750012292,13042814,21,1.82856085638,14042863,4,4.05655763101,150Process
Inserter-18:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in runProcess Inserter-39:Traceback (most recent call last):
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()Process Inserter-12:Traceback
(most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-19:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()Process Inserter-35:Traceback
(most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-41:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()Process Inserter-15:
>     self.run()Traceback (most recent call last):
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()43380,51,0.592962222588,16043556,17,0.85316223448,17043582,2,5.76186679877,18044110,52,0.574297892776,19044143,3,4.52656103625,20044799,65,0.46613632397,210
>
>
> 44828,2,5.20553808377,22044976,14,2.0302237675,230Process Inserter-31:Traceback (most
recent call last):Process Inserter-11:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-30:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrapProcess Inserter-27:Traceback (most recent call last):
>     self.run()
>   File "stress.py", line 167, in run
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()Process Inserter-46:Traceback
(most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()45324,34,0.361710271616,240Process
Inserter-44:Traceback (most recent call last):Process Inserter-21:Traceback (most recent call
last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-22:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()Process Inserter-49:Traceback
(most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()45394,7,2.14389359951,250Process
Inserter-10:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()45413,1,6.80838800731,26045836,42,0.212721344427,270Process
Inserter-13:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()45872,3,4.39313924313,28045899,2,2.80208562039,29046025,12,1.03094763226,30046036,1,6.91312122345,310Process
Inserter-5:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()46041,0,9.6405497551,32046937,89,0.210266448291,33047675,73,0.0936150796368,34049555,188,0.0744233962069,35050292,73,0.0938777962404,36051306,101,0.136950609717,37052649,134,0.0525897449286,38053247,59,0.229411067771,39053888,64,0.107282243541,40054971,108,0.0673501024836,410Process
Inserter-28:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-3:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertProcess Inserter-16:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.te
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insertTimedOutException: TimedOutException()
>     raise result.teTimedOutException: TimedOutException()55637,66,0.174130035234,42055875,23,0.201759393476,43057258,138,0.0706880513258,44057522,26,0.182813448436,450Process
Inserter-25:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in runProcess Inserter-17:Traceback (most recent call last):
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()58338,81,0.102476425615,46058565,22,0.151250916956,47058695,13,0.266876726884,48059481,78,0.0896081578641,49059767,28,0.121027883116,50061782,201,0.0350987749123,51066830,504,0.00852425800262,52067753,92,0.0383770502014,53067789,3,1.9088866512,54067789,0,NAN,550Process
Inserter-50:Process Inserter-40:Traceback (most recent call last):Traceback (most recent call
last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in runProcess Inserter-14:
>     self.run()
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "stress.py", line 167, in run
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insertTraceback (most recent call last):
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()67902,11,0.370109583424,560Process
Inserter-48:Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1_-py2.5-linux-x86_64.egg/multiprocessing/process.py",
line 236, in _bootstrap
>     self.run()
>   File "stress.py", line 167, in run
>     self.cclient.batch_insert('Keyspace1', key, cfmap, ConsistencyLevel.ONE)
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
574, in batch_insert
>     self.recv_batch_insert()
>   File "/home/live/dir/servers/cassandra/contrib/py_stress/cassandra/Cassandra.py", line
602, in recv_batch_insert
>     raise result.teTimedOutException: TimedOutException()67903,0,6.85239100456,57068002,9,0.13895741376,58068003,0,6.84438991547,590
>
> Extra information:
>
> 1st no clients were connected when it was run
> 2nd run, stress test was conducted when there were 50+ clients doing 7k read, 3-4k write
operations
> 3rd was run after clients were killed as problems showed up
> After 3rd run looked at tpstats for seed-node:Pool Name                    Active   Pending
     CompletedFILEUTILS-DELETE-POOL             0         0            442STREAM-STAGE   
                  0         0              0RESPONSE-STAGE                    0         0
       1427177ROW-READ-STAGE                    0         0        3705481LB-OPERATIONS  
                  0         0              0COMMITLOG                         1         0
       5133614MESSAGE-DESERIALIZER-POOL         0         0        3456220GMFD           
                  0         0          34882LB-TARGET                         0         0
             0CONSISTENCY-MANAGER               0         0              0ROW-MUTATION-STAGE
               0         0        5452268MESSAGE-STREAMING-POOL            0         0   
          0LOAD-BALANCER-STAGE               0         0              0FLUSH-SORTER-POOL 
               0         0            118MEMTABLE-POST-FLUSHER             0         0   
        118FLUSH-WRITER-POOL                 0         0            118AE-SERVICE-STAGE  
               0         0              1HINTED-HANDOFF-POOL               0         0   
         41
>
>


-- 
http://mixpanel.com
Blog: http://blog.mixpanel.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message