I think a degration could be normal because your cluster is in a degraded state when a node is down.

With a replication_factor of 3 and with a 3 nodes cluster, each data you write is replicated on each node. As One node is down, when writing, it's impossible to send a replica on the down node and hints are send : http://www.datastax.com/docs/1.0/dml/about_writes#hinted-handoff-writes
And it could be more expensive to achieve QUORUM when you read in that context.

It may be one explanation. You can turn cassandra log into debug level to see what happen when when there is a down node.

2011/10/19 Yang <teddyyyy123@gmail.com>


sorry forgot this important info

On Oct 19, 2011 11:31 AM, "Jérémy SEVELLEC" <jsevellec@gmail.com> wrote:
Hi, what is your replication_factor?

2011/10/19 Yang <teddyyyy123@gmail.com>
I'm using a cassandra version compiled from 1.0.0  github HEAD.

I have 3 nodes, A B and C,  on node A I run a client, which talks only
to B as the coordinator.

the performance is pretty good, a QUORUM read+write  takes < 10ms.

but then I shutdown C, quickly the performance starts to degrade, and
QUORUM read+write time steadily increase to about 300ms.

if I shutdown A and keep C, I observe the same effect.

I understand that "2 out of 3" is going to give you faster response
than "2 out of 2", but  the difference should not be that dramatic as
10ms vs 300ms.

any possible reasons for this?(or how to debug this?)