cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: tpstats confusion
Date Sat, 13 Aug 2011 03:46:33 GMT
Turn the logging up to DEBUG, if you want to you can do it just for org.apache.cassandra.service.StorageProxy
, and look for…

* forwarding counter update of key… when the coordinator is forwarding the counter update
* insert writing local & replicate…  when a counter mutation is applied locally 
* insert writing key… when the ROW is been sent

btw I was wrong about the sort by proximity. The default SimpleSnitch does no sorting, so
the endpoint order is the token order. The DynamicSnitch will re-order according to the observed
latency. 

Hope that helps. 
Aaron

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 13 Aug 2011, at 12:52, Ian Danforth wrote:

> Thanks! This would make sense but I'm not sure it maps to what I'm seeing. Specifically
if I start processes that write to each of the three nodes, so I know there is a constant
stream of writes to each node, instead of seeing replicate on write on all nodes, I see it
only on a single node.
> 
> It appears that all increments are being routed to a single node which then has to do
all replication. However I don't have a good way to see the write flow
> 
> Connection made to node: XXX
> Write sent to node: XXX
> Replicas sent to nodes: XXX, XXX
> 
> If I could see that I'd have a lot more confidence about what's going on.
> 
> Ian
> 
> On Thu, Aug 11, 2011 at 6:38 PM, aaron morton <aaron@thelastpickle.com> wrote:
> I've not checked the code but from memory when the nodes are ordered in proximity to
the coordinator the local node is always first if it's in the replica set. So with RF=3 and
N=3 the closest node is always the local one. 
> 
> Cheers
>  
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 12 Aug 2011, at 10:10, Ian Danforth wrote:
> 
>> I'm writing at QUORUM though and (pardon me for being dense) it would seem that the
first replica shouldn't always be on the same server if I'm using RP. 
>> 
>> I very much appreciate your time, I'm sure there is something basic that just hasn't
clicked.
>> 
>> Ian
>> 
>> On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>> https://issues.apache.org/jira/browse/CASSANDRA-2889
>> 
>> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <idanforth@numenta.com> wrote:
>> > I don't think so, perhaps more succinctly, why would ROW actions only be
>> > performed on a single node?
>> > Ian
>> >
>> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>> >>
>> >> are you i/o bound?  I believe counter ROW needs to perform a read of
>> >> the old value.
>> >>
>> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <idanforth@numenta.com>
>> >> wrote:
>> >> > Hello all!
>> >> > Thanks for taking the time to read this, I'm new to Cassandra and really
>> >> > want to get it working :)
>> >> > Below you'll find the output from tpstats on the three nodes in my
.8.1
>> >> > cluster. As you'll note from ring (also below), my cluster appears
>> >> > balanced.
>> >> > However, please note that NODE1 is the only node that has completed
any
>> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
>> >> > RoW
>> >> > and seen a single node in my various test clusters get hugely backed
up
>> >> > on
>> >> > this stage to the point where it slows all writes to a crawl.
>> >> > I'm using counters in a column family defined like this:
>> >> > ==== KEYSPACE/CF ====
>> >> > create keyspace Keyspace1
>> >> >     with strategy_options=[{replication_factor:3}]
>> >> >     and placement_strategy =
>> >> > 'org.apache.cassandra.locator.SimpleStrategy';
>> >> > use Keyspace1;
>> >> > create column family TwitterTest
>> >> >     with default_validation_class = CounterColumnType
>> >> >     and replicate_on_write = true;
>> >> > ====
>> >> >
>> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
>> >> > backed
>> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was suggested
>> >> > elsewhere that ANY or ONE could overwhelm the system.
>> >> >
>> >> > I hope I'm just doing something stupid and one you experienced folks
can
>> >> > point out my error.
>> >> > Thanks in advance!
>> >> > Ian
>> >> > ==== MORE DATA ====
>> >> > Address         DC          Rack        Status State   Load
>> >> >  Owns
>> >> >    Token
>> >> >
>> >> >    113427455640312821154458202477256070484
>> >> > 10.87.9.57      datacenter1 rack1       Up     Normal  193.86 MB
>> >> > 33.33%  0
>> >> > 10.86.223.32    datacenter1 rack1       Up     Normal  191.1 MB
>> >> >  33.33%  56713727820156410577229101238628035242
>> >> > 10.86.229.24    datacenter1 rack1       Up     Normal  193.63 MB
>> >> > 33.33%  113427455640312821154458202477256070484
>> >> >
>> >> >
>> >> >
>> >> > SEED
>> >> > Pool Name                    Active   Pending      Completed
>> >> > ReadStage                         0         0              7
>> >> > RequestResponseStage              0         0        1479839
>> >> > MutationStage                     0         0        1479908
>> >> > ReadRepairStage                   0         0              0
>> >> > ReplicateOnWriteStage             0         0              0
>> >> > GossipStage                       0         0           5176
>> >> > AntiEntropyStage                  0         0              0
>> >> > MigrationStage                    0         0            128
>> >> > MemtablePostFlusher               0         0             28
>> >> > StreamStage                       0         0              0
>> >> > FlushWriter                       0         0             28
>> >> > MiscStage                         0         0              2
>> >> > FlushSorter                       0         0              0
>> >> > InternalResponseStage             0         0            363
>> >> > HintedHandoff                     0         0              0
>> >> > NODE1
>> >> > Pool Name                    Active   Pending      Completed
>> >> > ReadStage                         0         0              7
>> >> > RequestResponseStage              0         0        2960208
>> >> > MutationStage                     0         0        1480104
>> >> > ReadRepairStage                   0         0              0
>> >> > ReplicateOnWriteStage             0         0        1480104
>> >> > GossipStage                       0         0           4151
>> >> > AntiEntropyStage                  0         0              0
>> >> > MigrationStage                    0         0            128
>> >> > MemtablePostFlusher               0         0             31
>> >> > StreamStage                       0         0              8
>> >> > FlushWriter                       0         0             31
>> >> > MiscStage                         0         0              8
>> >> > FlushSorter                       0         0              0
>> >> > InternalResponseStage             0         0              1
>> >> > HintedHandoff                     0         0              0
>> >> > NODE2
>> >> > Pool Name                    Active   Pending      Completed
>> >> > ReadStage                         0         0              7
>> >> > RequestResponseStage              0         0              0
>> >> > MutationStage                     0         0        1481530
>> >> > ReadRepairStage                   0         0              0
>> >> > ReplicateOnWriteStage             0         0              0
>> >> > GossipStage                       0         0           4201
>> >> > AntiEntropyStage                  0         0              0
>> >> > MigrationStage                    0         0            128
>> >> > MemtablePostFlusher               0         0             31
>> >> > StreamStage                       0         0              6
>> >> > FlushWriter                       0         0             31
>> >> > MiscStage                         0         0              6
>> >> > FlushSorter                       0         0              0
>> >> > InternalResponseStage             0         0              1
>> >> > HintedHandoff                     0         0              0
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>> 
>> 
>> 
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>> 
> 
> 


Mime
View raw message