Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6E6BB70D6 for ; Fri, 12 Aug 2011 01:39:30 +0000 (UTC) Received: (qmail 50190 invoked by uid 500); 12 Aug 2011 01:39:27 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 50071 invoked by uid 500); 12 Aug 2011 01:39:24 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50058 invoked by uid 99); 12 Aug 2011 01:39:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Aug 2011 01:39:22 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a58.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Aug 2011 01:39:17 +0000 Received: from homiemail-a58.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a58.g.dreamhost.com (Postfix) with ESMTP id DC2377D8063 for ; Thu, 11 Aug 2011 18:38:55 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=LazlaqGsSO FzyfJ0IVyIF4u9Tl9NQaEyVWwio7JV3Y0fhPwojhtUP8VBxRco/Tlo0cfAHxNGAk SWq48fTDiTgAHcXedicBgdBSm3xXgqWq+iV3k3W+XJ6uyzR15x/ORY7aStqNnrGb JMiEDIJ21qifUsefXIwUSWrBeAPIo1j/c= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=mo3xOntTJlpaq0dZ 1Js+qAMWLbg=; b=0OOJLFPy2J3cyVoIeIbVp4gG6lZg3j/+Npaq4EfiZ4IE7lji UdvRnzsO+kzmWfQUn28DPhTMAvo0mIT+BaCQdZpRLPiYxrlCXGadaCEWjfb4V4bp B9JMok5KtOJSc9o/5J6B+3YfxikyUd+OGbpcgcVbBsgWNWtB5ZW5Jnn7dhs= Received: from [10.0.1.150] (219-89-250-213.adsl.xtra.co.nz [219.89.250.213]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a58.g.dreamhost.com (Postfix) with ESMTPSA id EC4F77D805B for ; Thu, 11 Aug 2011 18:38:54 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1244.3) Content-Type: multipart/alternative; boundary="Apple-Mail=_663C123E-DEDA-418F-B9D5-F84CF5A86713" Subject: Re: tpstats confusion Date: Fri, 12 Aug 2011 13:38:55 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1244.3) --Apple-Mail=_663C123E-DEDA-418F-B9D5-F84CF5A86713 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 I've not checked the code but from memory when the nodes are ordered in = proximity to the coordinator the local node is always first if it's in = the replica set. So with RF=3D3 and N=3D3 the closest node is always the = local one.=20 Cheers =20 ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 12 Aug 2011, at 10:10, Ian Danforth wrote: > I'm writing at QUORUM though and (pardon me for being dense) it would = seem that the first replica shouldn't always be on the same server if = I'm using RP.=20 >=20 > I very much appreciate your time, I'm sure there is something basic = that just hasn't clicked. >=20 > Ian >=20 > On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis = wrote: > https://issues.apache.org/jira/browse/CASSANDRA-2889 >=20 > On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth = wrote: > > I don't think so, perhaps more succinctly, why would ROW actions = only be > > performed on a single node? > > Ian > > > > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis = wrote: > >> > >> are you i/o bound? I believe counter ROW needs to perform a read = of > >> the old value. > >> > >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth = > >> wrote: > >> > Hello all! > >> > Thanks for taking the time to read this, I'm new to Cassandra and = really > >> > want to get it working :) > >> > Below you'll find the output from tpstats on the three nodes in = my .8.1 > >> > cluster. As you'll note from ring (also below), my cluster = appears > >> > balanced. > >> > However, please note that NODE1 is the only node that has = completed any > >> > ReplicateOnWriteStage actions. I've been having a lot of trouble = with > >> > RoW > >> > and seen a single node in my various test clusters get hugely = backed up > >> > on > >> > this stage to the point where it slows all writes to a crawl. > >> > I'm using counters in a column family defined like this: > >> > =3D=3D=3D=3D KEYSPACE/CF =3D=3D=3D=3D > >> > create keyspace Keyspace1 > >> > with strategy_options=3D[{replication_factor:3}] > >> > and placement_strategy =3D > >> > 'org.apache.cassandra.locator.SimpleStrategy'; > >> > use Keyspace1; > >> > create column family TwitterTest > >> > with default_validation_class =3D CounterColumnType > >> > and replicate_on_write =3D true; > >> > =3D=3D=3D=3D > >> > > >> > With any given batch_insert() I'm incrementing ~17,000 counters. = I've > >> > backed > >> > off a bit and started using ConsistencyLevel.QUORUM, as it was = suggested > >> > elsewhere that ANY or ONE could overwhelm the system. > >> > > >> > I hope I'm just doing something stupid and one you experienced = folks can > >> > point out my error. > >> > Thanks in advance! > >> > Ian > >> > =3D=3D=3D=3D MORE DATA =3D=3D=3D=3D > >> > Address DC Rack Status State Load > >> > Owns > >> > Token > >> > > >> > 113427455640312821154458202477256070484 > >> > 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB > >> > 33.33% 0 > >> > 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB > >> > 33.33% 56713727820156410577229101238628035242 > >> > 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB > >> > 33.33% 113427455640312821154458202477256070484 > >> > > >> > > >> > > >> > SEED > >> > Pool Name Active Pending Completed > >> > ReadStage 0 0 7 > >> > RequestResponseStage 0 0 1479839 > >> > MutationStage 0 0 1479908 > >> > ReadRepairStage 0 0 0 > >> > ReplicateOnWriteStage 0 0 0 > >> > GossipStage 0 0 5176 > >> > AntiEntropyStage 0 0 0 > >> > MigrationStage 0 0 128 > >> > MemtablePostFlusher 0 0 28 > >> > StreamStage 0 0 0 > >> > FlushWriter 0 0 28 > >> > MiscStage 0 0 2 > >> > FlushSorter 0 0 0 > >> > InternalResponseStage 0 0 363 > >> > HintedHandoff 0 0 0 > >> > NODE1 > >> > Pool Name Active Pending Completed > >> > ReadStage 0 0 7 > >> > RequestResponseStage 0 0 2960208 > >> > MutationStage 0 0 1480104 > >> > ReadRepairStage 0 0 0 > >> > ReplicateOnWriteStage 0 0 1480104 > >> > GossipStage 0 0 4151 > >> > AntiEntropyStage 0 0 0 > >> > MigrationStage 0 0 128 > >> > MemtablePostFlusher 0 0 31 > >> > StreamStage 0 0 8 > >> > FlushWriter 0 0 31 > >> > MiscStage 0 0 8 > >> > FlushSorter 0 0 0 > >> > InternalResponseStage 0 0 1 > >> > HintedHandoff 0 0 0 > >> > NODE2 > >> > Pool Name Active Pending Completed > >> > ReadStage 0 0 7 > >> > RequestResponseStage 0 0 0 > >> > MutationStage 0 0 1481530 > >> > ReadRepairStage 0 0 0 > >> > ReplicateOnWriteStage 0 0 0 > >> > GossipStage 0 0 4201 > >> > AntiEntropyStage 0 0 0 > >> > MigrationStage 0 0 128 > >> > MemtablePostFlusher 0 0 31 > >> > StreamStage 0 0 6 > >> > FlushWriter 0 0 31 > >> > MiscStage 0 0 6 > >> > FlushSorter 0 0 0 > >> > InternalResponseStage 0 0 1 > >> > HintedHandoff 0 0 0 > >> > >> > >> > >> -- > >> Jonathan Ellis > >> Project Chair, Apache Cassandra > >> co-founder of DataStax, the source for professional Cassandra = support > >> http://www.datastax.com > > > > >=20 >=20 >=20 > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >=20 --Apple-Mail=_663C123E-DEDA-418F-B9D5-F84CF5A86713 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1 I've = not checked the code but from memory when the nodes are ordered in = proximity to the coordinator the local node is always first if it's in = the replica set. So with RF=3D3 and N=3D3 the closest node is always the = local one. 

Cheers
 
http://www.thelastpickle.com

On 12 Aug 2011, at 10:10, Ian Danforth wrote:

I'm = writing at QUORUM though and (pardon me for being dense) it would seem = that the first replica shouldn't always be on the same server if I'm = using RP. 

I very much appreciate your time, I'm = sure there is something basic that just hasn't clicked.

Ian

On Thu, Aug = 11, 2011 at 2:56 PM, Jonathan Ellis <jbellis@gmail.com> = wrote:
https://issues.apache.org/jira/browse/CASSANDRA-2889=

On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <idanforth@numenta.com> = wrote:
> I don't think so, perhaps more succinctly, why would ROW actions = only be
> performed on a single node?
> Ian
>
> On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> are you i/o bound?  I believe counter ROW needs to perform = a read of
>> the old value.
>>
>> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <idanforth@numenta.com>
>> wrote:
>> > Hello all!
>> > Thanks for taking the time to read this, I'm new to = Cassandra and really
>> > want to get it working :)
>> > Below you'll find the output from tpstats on the three = nodes in my .8.1
>> > cluster. As you'll note from ring (also below), my cluster = appears
>> > balanced.
>> > However, please note that NODE1 is the only node that has = completed any
>> > ReplicateOnWriteStage actions. I've been having a lot of = trouble with
>> > RoW
>> > and seen a single node in my various test clusters get = hugely backed up
>> > on
>> > this stage to the point where it slows all writes to a = crawl.
>> > I'm using counters in a column family defined like = this:
>> > =3D=3D=3D=3D KEYSPACE/CF =3D=3D=3D=3D
>> > create keyspace Keyspace1
>> >     with = strategy_options=3D[{replication_factor:3}]
>> >     and placement_strategy =3D
>> > 'org.apache.cassandra.locator.SimpleStrategy';
>> > use Keyspace1;
>> > create column family TwitterTest
>> >     with default_validation_class =3D = CounterColumnType
>> >     and replicate_on_write =3D true;
>> > =3D=3D=3D=3D
>> >
>> > With any given batch_insert() I'm incrementing ~17,000 = counters. I've
>> > backed
>> > off a bit and started using ConsistencyLevel.QUORUM, as it = was suggested
>> > elsewhere that ANY or ONE could overwhelm the system.
>> >
>> > I hope I'm just doing something stupid and one you = experienced folks can
>> > point out my error.
>> > Thanks in advance!
>> > Ian
>> > =3D=3D=3D=3D MORE DATA =3D=3D=3D=3D
>> > Address         DC     =      Rack        Status State   = Load
>> >  Owns
>> >    Token
>> >
>> >    113427455640312821154458202477256070484
>> > 10.87.9.57      datacenter1 rack1   =     Up     Normal  193.86 MB
>> > 33.33%  0
>> > 10.86.223.32    datacenter1 rack1     =   Up     Normal  191.1 MB
>> >  33.33% =  56713727820156410577229101238628035242
>> > 10.86.229.24    datacenter1 rack1     =   Up     Normal  193.63 MB
>> > 33.33%  113427455640312821154458202477256070484
>> >
>> >
>> >
>> > SEED
>> > Pool Name               =      Active   Pending     =  Completed
>> > ReadStage               =           0         0 =              7
>> > RequestResponseStage           =    0         0       =  1479839
>> > MutationStage             =         0         0   =      1479908
>> > ReadRepairStage             =       0         0     =          0
>> > ReplicateOnWriteStage           =   0         0         =      0
>> > GossipStage             =           0         0 =           5176
>> > AntiEntropyStage             =      0         0       =        0
>> > MigrationStage             =        0         0     =        128
>> > MemtablePostFlusher           =     0         0       =       28
>> > StreamStage             =           0         0 =              0
>> > FlushWriter             =           0         0 =             28
>> > MiscStage               =           0         0 =              2
>> > FlushSorter             =           0         0 =              0
>> > InternalResponseStage           =   0         0         =    363
>> > HintedHandoff             =         0         0   =            0
>> > NODE1
>> > Pool Name               =      Active   Pending     =  Completed
>> > ReadStage               =           0         0 =              7
>> > RequestResponseStage           =    0         0       =  2960208
>> > MutationStage             =         0         0   =      1480104
>> > ReadRepairStage             =       0         0     =          0
>> > ReplicateOnWriteStage           =   0         0       =  1480104
>> > GossipStage             =           0         0 =           4151
>> > AntiEntropyStage             =      0         0       =        0
>> > MigrationStage             =        0         0     =        128
>> > MemtablePostFlusher           =     0         0       =       31
>> > StreamStage             =           0         0 =              8
>> > FlushWriter             =           0         0 =             31
>> > MiscStage               =           0         0 =              8
>> > FlushSorter             =           0         0 =              0
>> > InternalResponseStage           =   0         0         =      1
>> > HintedHandoff             =         0         0   =            0
>> > NODE2
>> > Pool Name               =      Active   Pending     =  Completed
>> > ReadStage               =           0         0 =              7
>> > RequestResponseStage           =    0         0         =      0
>> > MutationStage             =         0         0   =      1481530
>> > ReadRepairStage             =       0         0     =          0
>> > ReplicateOnWriteStage           =   0         0         =      0
>> > GossipStage             =           0         0 =           4201
>> > AntiEntropyStage             =      0         0       =        0
>> > MigrationStage             =        0         0     =        128
>> > MemtablePostFlusher           =     0         0       =       31
>> > StreamStage             =           0         0 =              6
>> > FlushWriter             =           0         0 =             31
>> > MiscStage               =           0         0 =              6
>> > FlushSorter             =           0         0 =              0
>> > InternalResponseStage           =   0         0         =      1
>> > HintedHandoff             =         0         0   =            0
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra = support
>> http://www.datastax.com
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra = support
http://www.datastax.com


= --Apple-Mail=_663C123E-DEDA-418F-B9D5-F84CF5A86713--