Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BE284FF05 for ; Thu, 21 Mar 2013 17:22:37 +0000 (UTC) Received: (qmail 59612 invoked by uid 500); 21 Mar 2013 17:22:35 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 59508 invoked by uid 500); 21 Mar 2013 17:22:35 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 59477 invoked by uid 99); 21 Mar 2013 17:22:34 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Mar 2013 17:22:34 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a94.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 21 Mar 2013 17:22:30 +0000 Received: from homiemail-a94.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a94.g.dreamhost.com (Postfix) with ESMTP id C16BD38A074 for ; Thu, 21 Mar 2013 10:22:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=qSyeEQW/W9uKPH6BAo/XoMcd9F Y=; b=XdFbwirLp56FHWny3dYd3wYBS+/ateUMpqd/O5KCPM9Dc/PkbWb93Slt8o 9tzmdI1XFLq2w7iwt+olDA4GhZHozPybJMls/+2mpm2mzkyl3pb9FQ7tCZZwtf3/ yeZTsGv2QaUSAunob/r5vsoQs7kALXjZvKGizsou6POoVUdx4= Received: from [172.16.1.8] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a94.g.dreamhost.com (Postfix) with ESMTPSA id E2FF938A06F for ; Thu, 21 Mar 2013 10:22:08 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_F789CE3B-F97C-4389-8DFE-D0F74A1A124B" Message-Id: <36623AED-B659-4CCF-9427-FF8769F48F39@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Question regarding multi datacenter and LOCAL_QUORUM Date: Fri, 22 Mar 2013 06:22:07 +1300 References: <310FEAA2F0C52A4F9744425E3C10CF1606525E@GENSJZMBX02.msg.int.genesyslab.com> <310FEAA2F0C52A4F9744425E3C10CF160653AD@GENSJZMBX02.msg.int.genesyslab.com> <310FEAA2F0C52A4F9744425E3C10CF1606541C@GENSJZMBX02.msg.int.genesyslab.com> <310FEAA2F0C52A4F9744425E3C10CF1606546F@GENSJZMBX02.msg.int.genesyslab.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_F789CE3B-F97C-4389-8DFE-D0F74A1A124B Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > DEBUG [Thrift:1] 2013-03-19 00:00:53,313 ReadCallback.java (line 79) = Blockfor is 2; setting up requests to = /xx.yy.zz.146,/xx.yy.zz.143,/xx.yy.zz.145 > DEBUG [Thrift:1] 2013-03-19 00:00:53,334 CassandraServer.java (line = 306) get_slice > DEBUG [Thrift:1] 2013-03-19 00:00:53,334 ReadCallback.java (line 79) = Blockfor is 2; setting up requests to /xx.yy.zz.146,/xx.yy.zz.143 > DEBUG [Thrift:1] 2013-03-19 00:00:53,366 CassandraServer.java (line = 306) get_slice > DEBUG [Thrift:1] 2013-03-19 00:00:53,367 ReadCallback.java (line 79) = Blockfor is 2; setting up requests to = /xx.yy.zz.146,/xx.yy.zz.143,/xx.yy.zz.145 This is Read Repair, as controlled by the read_repaur_chance and = dclocal_read_repair_chance CF settings, in action.=20 "Blockfor" is how many nodes the read operation is going to wait for. = When the number of nodes in the request is more than blockfor it means = Read Repair is active, we are reading from all UP nodes and will repair = any detected differences in the background. Your read is waiting for 2 = nodes to respond only (including the one we ask for the data.) The odd thing here is that there are only 3 replicas nodes. Are you = using the Network Topology Strategy ? If so I would expect there to be 6 = nodes in the the request with RR, 3 in each DC.=20 Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 21/03/2013, at 12:38 PM, Tyler Hobbs wrote: >=20 > On Wed, Mar 20, 2013 at 3:18 PM, Tycen Stafford = wrote: > I don=92t think that=92s correct for a mult-dc ring, but you=92ll want = to hear a final answer from someone more authoritative. I could easily = be wrong. Try using the built in token generating tool = (token-generator) =96 I don=92t seem to have it on my hosts (1.1.6 also) = so I can=92t confirm. I used the tokentoolv2.py tool (from here = http://www.datastax.com/docs/1.0/initialize/token_generation) and got = the following (which looks to me evenly spaced and not using offsets): >=20 > tstafford@tycen-linux:Cassandra$ ./tokentoolv2.py 3 3 >=20 > { >=20 > "0": { >=20 > "0": 0, >=20 > "1": 56713727820156410577229101238628035242, >=20 > "2": 113427455640312821154458202477256070485 >=20 > }, >=20 > "1": { >=20 > "0": 28356863910078205288614550619314017621, >=20 > "1": 85070591730234615865843651857942052863, >=20 > "2": 141784319550391026443072753096570088106 >=20 > } >=20 > } >=20 >=20 > For multi-DC clusters, the only requirement for a balanced cluster is = that all tokens within a DC must be balanced; you can basically treat = each DC as a separate ring (as long as your tokens don't line up = exactly). So, either using an offset for the second DC or evenly = spacing all nodes is acceptable. >=20 > --=20 > Tyler Hobbs > DataStax --Apple-Mail=_F789CE3B-F97C-4389-8DFE-D0F74A1A124B Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 DEBUG [Thrift:1] 2013-03-19 00:00:53,313 = ReadCallback.java (line 79) Blockfor is 2; setting up requests to = /xx.yy.zz.146,/xx.yy.zz.143,/xx.yy.zz.145
DEBUG [Thrift:1] 2013-03-19 = 00:00:53,334 CassandraServer.java (line 306) get_slice
DEBUG = [Thrift:1] 2013-03-19 00:00:53,334 ReadCallback.java (line 79) Blockfor = is 2; setting up requests to /xx.yy.zz.146,/xx.yy.zz.143
DEBUG = [Thrift:1] 2013-03-19 00:00:53,366 CassandraServer.java (line 306) = get_slice
DEBUG [Thrift:1] 2013-03-19 00:00:53,367 ReadCallback.java = (line 79) Blockfor is 2; setting up requests to = /xx.yy.zz.146,/xx.yy.zz.143,/xx.yy.zz.145
This is Read = Repair, as controlled by the read_repaur_chance = and dclocal_read_repair_chance CF settings, in = action. 

"Blockfor" is how many nodes the read = operation is going to wait for. When the number of nodes in the request = is more than blockfor it means Read Repair is active, we are reading = from all UP nodes and will repair any detected differences in the = background. Your read is waiting for 2 nodes to respond only (including = the one we ask for the data.)

The odd thing = here is that there are only 3 replicas nodes. Are you using the Network = Topology Strategy ? If so I would expect there to be 6 nodes in the the = request with RR, 3 in each = DC. 

Cheers


Aaron = Morton
Freelance Cassandra = Consultant
New Zealand


http://www.thelastpickle.com

On 21/03/2013, at 12:38 PM, Tyler Hobbs <tyler@datastax.com> = wrote:


On Wed, Mar 20, 2013 at 3:18 PM, Tycen Stafford = <tstafford@medio.com> wrote:

I don=92t think that=92s= correct for a mult-dc ring, but you=92ll want to hear a final answer = from someone more authoritative.  I could easily be wrong.  = Try using the built in token generating tool (token-generator) =96 I = don=92t seem to have it on my hosts (1.1.6 also) so I can=92t = confirm.  I used the tokentoolv2.py tool (from here http://www.datastax.com/docs/1.0/initialize/token_genera= tion) and got the following (which looks to me evenly spaced and not = using offsets):

tstafford@tycen-linux:Cassandra$ ./tokentoolv2.py 3 = 3

{

    = "0": {

        "0": = 0,

        "1= ": 56713727820156410577229101238628035242,

        "2= ": 113427455640312821154458202477256070485

    }, =

    "1": = {

        "0": = 28356863910078205288614550619314017621,

        "1= ": 85070591730234615865843651857942052863,

        "2= ": 141784319550391026443072753096570088106

    = }

}


For multi-DC = clusters, the only requirement for a balanced cluster is that all tokens = within a DC must be balanced; you can basically treat each DC as a = separate ring (as long as your tokens don't line up exactly).  So, = either using an offset for the second DC or evenly spacing all nodes is = acceptable.

--
Tyler Hobbs
DataStax
=

= --Apple-Mail=_F789CE3B-F97C-4389-8DFE-D0F74A1A124B--