incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daning Wang <dan...@netseer.com>
Subject Re: Multiple data center performance
Date Tue, 11 Jun 2013 21:34:16 GMT
It is counter caused the problem. counter will replicate to all replicas
during write regardless the consistency level.

In our case. we don't need to sync the counter across the center. so moving
counter to new keyspace and all the replica in one center solved problem.

There is option replicate_on_write on table. If you turn that off for
counter might have better performance. but you are on high risk to lose
data and create inconsistency. I did not try this option.

Daning


On Sat, Jun 8, 2013 at 6:53 AM, srmore <comomore@gmail.com> wrote:

> I am seeing the similar behavior, in my case I have 2 nodes in each
> datacenter and one node always has high latency (equal to the latency
> between the two datacenters). When one of the datacenters is shutdown the
> latency drops.
>
> I am curious to know whether anyone else has these issues and if yes how
> did to get around it.
>
> Thanks !
>
>
> On Fri, Jun 7, 2013 at 11:49 PM, Daning Wang <daning@netseer.com> wrote:
>
>> We have deployed multi-center but got performance issue. When the nodes
>> on other center are up, the read response time from clients is 4 or 5 times
>> higher. when we take those nodes down, the response time becomes
>> normal(compare to the time before we changed to multi-center).
>>
>> We have high volume on the cluster, the consistency level is one for
>> read. so my understanding is most of traffic between data center should be
>> read repair. but seems that could not create much delay.
>>
>> What could cause the problem? how to debug this?
>>
>> Here is the keyspace,
>>
>> [default@dsat] describe dsat;
>> Keyspace: dsat:
>>   Replication Strategy:
>> org.apache.cassandra.locator.NetworkTopologyStrategy
>>   Durable Writes: true
>>     Options: [dc2:1, dc1:3]
>>   Column Families:
>>     ColumnFamily: categorization_cache
>>
>>
>> Ring
>>
>> Datacenter: dc1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns (effective)  Host ID
>>                         Rack
>> UN  xx.xx.xx..111       59.2 GB    256     37.5%
>> 4d6ed8d6-870d-4963-8844-08268607757e  rac1
>> DN  xx.xx.xx..121       99.63 GB   256     37.5%
>> 9d0d56ce-baf6-4440-a233-ad6f1d564602  rac1
>> UN  xx.xx.xx..120       66.32 GB   256     37.5%
>> 0fd912fb-3187-462b-8c8a-7d223751b649  rac1
>> UN  xx.xx.xx..118       63.61 GB   256     37.5%
>> 3c6e6862-ab14-4a8c-9593-49631645349d  rac1
>> UN  xx.xx.xx..117       68.16 GB   256     37.5%
>> ee6cdf23-d5e4-4998-a2db-f6c0ce41035a  rac1
>> UN  xx.xx.xx..116       32.41 GB   256     37.5%
>> f783eeef-1c51-4f91-ab7c-a60669816770  rac1
>> UN  xx.xx.xx..115       64.24 GB   256     37.5%
>> e75105fb-b330-4f40-aa4f-8e6e11838e37  rac1
>> UN  xx.xx.xx..112       61.32 GB   256     37.5%
>> 2547ee54-88dd-4994-a1ad-d9ba367ed11f  rac1
>> Datacenter: dc2
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address           Load       Tokens  Owns (effective)  Host ID
>>                         Rack
>> DN  xx.xx.xx.199    58.39 GB   256     50.0%
>> 6954754a-e9df-4b3c-aca7-146b938515d8  rac1
>> DN  xx.xx.xx..61      33.79 GB   256     50.0%
>> 91b8d510-966a-4f2d-a666-d7edbe986a1c  rac1
>>
>>
>> Thank you in advance,
>>
>> Daning
>>
>>
>

Mime
View raw message