Mailing-List: contact dev-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <506DD49E.7000506@mustardgrain.com>
Date: Thu, 04 Oct 2012 11:25:34 -0700
From: Kirk True <kirk@mustardgrain.com>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8;
 rv:15.0) Gecko/20120907 Thunderbird/15.0.1
MIME-Version: 1.0
To: dev@cassandra.apache.org
Subject: Expected behavior of number of nodes contacted during CL=QUORUM read
Content-Type: multipart/alternative;
 boundary="------------010609080805040607000906"

--------------010609080805040607000906
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi all,

Test scenario:

     4 nodes (.1, .2, .3, .4)
     RF=3
     CL=QUORUM
     1.1.2

I noticed that in ReadCallback's constructor, it determines the 
'blockfor' number of 2 for RF=3, CL=QUORUM.

According to the API page on the wiki[1] for reads at CL=QUORUM:

    Will query *all* replicas and return the record with the most recent
    timestamp once it has at least a majority of replicas (N / 2 + 1)
    reported.


However, in ReadCallback's constructor, it determines blockfor to be 2, 
then calls filterEndpoints. filterEndpoints is given a list of the three 
replicas, but at the very end of the method, the endpoint list to only 
two replicas. Those two replicas are then used in StorageProxy to 
execute the read/digest calls. So it ends up as 2 nodes, not all three 
as stated on the wiki.

In my test case, I kill a node and then immediately issue a query for a 
key that has a replica on the downed node. For the live nodes in the 
system, it doesn't immediately know that the other node is down yet. 
Rather than contacting *all* nodes as the wiki states, the coordinator 
contacts only two -- one of which is the downed node. Since it blocks 
for two, one of which is down, the query times out. Attempting the read 
again produces the same effect, even when trying different nodes as 
coordinators. I end up retrying a few times until the failure detectors 
on the live nodes realize that the node is down.

So, the end result is that if a client attempts to read a row that has a 
replica on a newly downed node, it will timeout repeatedly until the ~30 
seconds failure detector window has passed -- even though there are 
enough live replicas to satisfy the request. We basically have a 
scenario wherein a value is not retrievable for upwards of 30 seconds. 
The percentage of keys that exhibit this possibility shrinks as the ring 
grows, but it's still non-zero.

This doesn't seem right and I'm sure I'm missing something.

Thanks,
Kirk

[1] http://wiki.apache.org/cassandra/API

--------------010609080805040607000906--