incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Mutation dropped
Date Sun, 17 Feb 2013 17:26:29 GMT
You are hitting the maximum throughput on the cluster. 

The messages are dropped because the node fails to start processing them before rpc_timeout.


However the request is still a success because the client requested CL was achieved. 

Testing with RF 2 and CL 1 really just tests the disks on one local machine. Both nodes replicate
each row, and writes are sent to each replica, so the only thing the client is waiting on
is the local node to write to it's commit log. 

Testing with (and running in prod) RF3 and CL QUROUM is a more real world scenario. 

Cheers
 
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 15/02/2013, at 9:42 AM, Kanwar Sangha <kanwar@mavenir.com> wrote:

> Hi – Is there a parameter which can be tuned to prevent the mutations from being dropped
? Is this logic correct ?
>  
> Node A and B with RF=2, CL =1. Load balanced between the two.
>  
> --  Address           Load       Tokens  Owns (effective)  Host ID                  
            Rack
> UN  10.x.x.x       746.78 GB  256     100.0%            dbc9e539-f735-4b0b-8067-b97a85522a1a
 rack1
> UN  10.x.x.x       880.77 GB  256     100.0%            95d59054-be99-455f-90d1-f43981d3d778
 rack1
>  
> Once we hit a very high TPS (around 50k/sec of inserts), the nodes start falling behind
and we see the mutation dropped messages. But there are no failures on the client. Does that
mean other node is not able to persist the replicated data ? Is there some timeout associated
with replicated data persistence ?
>  
> Thanks,
> Kanwar
>  
>  
>  
>  
>  
>  
>  
> From: Kanwar Sangha [mailto:kanwar@mavenir.com] 
> Sent: 14 February 2013 09:08
> To: user@cassandra.apache.org
> Subject: Mutation dropped
>  
> Hi – I am doing a load test using YCSB across 2 nodes in a cluster and seeing a lot
of mutation dropped messages.  I understand that this is due to the replica not being written
to the
> other node ? RF = 2, CL =1.
>  
> From the wiki -
> For MUTATION messages this means that the mutation was not applied to all replicas it
was sent to. The inconsistency will be repaired by Read Repair or Anti Entropy Repair
>  
> Thanks,
> Kanwar
>  


Mime
View raw message