cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Danforth <idanfo...@numenta.com>
Subject Re: tpstats confusion
Date Sat, 13 Aug 2011 00:52:00 GMT
Thanks! This would make sense but I'm not sure it maps to what I'm seeing.
Specifically if I start processes that write to each of the three nodes, so
I know there is a constant stream of writes to each node, instead of seeing
replicate on write on all nodes, I see it only on a single node.

It appears that all increments are being routed to a single node which then
has to do all replication. However I don't have a good way to see the write
flow

Connection made to node: XXX
Write sent to node: XXX
Replicas sent to nodes: XXX, XXX

If I could see that I'd have a lot more confidence about what's going on.

Ian

On Thu, Aug 11, 2011 at 6:38 PM, aaron morton <aaron@thelastpickle.com>wrote:

>  I've not checked the code but from memory when the nodes are ordered in
> proximity to the coordinator the local node is always first if it's in the
> replica set. So with RF=3 and N=3 the closest node is always the local one.
>
>
>  Cheers
>
>    -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
>  On 12 Aug 2011, at 10:10, Ian Danforth wrote:
>
> I'm writing at QUORUM though and (pardon me for being dense) it would seem
> that the first replica shouldn't always be on the same server if I'm using
> RP.
>
>  I very much appreciate your time, I'm sure there is something basic that
> just hasn't clicked.
>
>  Ian
>
> On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>
>> https://issues.apache.org/jira/browse/CASSANDRA-2889
>>
>> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <idanforth@numenta.com>
>> wrote:
>> > I don't think so, perhaps more succinctly, why would ROW actions only be
>> > performed on a single node?
>> > Ian
>> >
>> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jbellis@gmail.com>
>> wrote:
>> >>
>> >> are you i/o bound?  I believe counter ROW needs to perform a read of
>> >> the old value.
>> >>
>> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <idanforth@numenta.com>
>> >> wrote:
>> >> > Hello all!
>> >> > Thanks for taking the time to read this, I'm new to Cassandra and
>> really
>> >> > want to get it working :)
>> >> > Below you'll find the output from tpstats on the three nodes in my
>> .8.1
>> >> > cluster. As you'll note from ring (also below), my cluster appears
>> >> > balanced.
>> >> > However, please note that NODE1 is the only node that has completed
>> any
>> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
>> >> > RoW
>> >> > and seen a single node in my various test clusters get hugely backed
>> up
>> >> > on
>> >> > this stage to the point where it slows all writes to a crawl.
>> >> > I'm using counters in a column family defined like this:
>> >> > ==== KEYSPACE/CF ====
>> >> > create keyspace Keyspace1
>> >> >     with strategy_options=[{replication_factor:3}]
>> >> >     and placement_strategy =
>> >> > 'org.apache.cassandra.locator.SimpleStrategy';
>> >> > use Keyspace1;
>> >> > create column family TwitterTest
>> >> >     with default_validation_class = CounterColumnType
>> >> >     and replicate_on_write = true;
>> >> > ====
>> >> >
>> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
>> >> > backed
>> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was
>> suggested
>> >> > elsewhere that ANY or ONE could overwhelm the system.
>> >> >
>> >> > I hope I'm just doing something stupid and one you experienced folks
>> can
>> >> > point out my error.
>> >> > Thanks in advance!
>> >> > Ian
>> >> > ==== MORE DATA ====
>> >> > Address         DC          Rack        Status State   Load
>> >> >  Owns
>> >> >    Token
>> >> >
>> >> >    113427455640312821154458202477256070484
>> >> > 10.87.9.57      datacenter1 rack1       Up     Normal  193.86 MB
>> >> > 33.33%  0
>> >> > 10.86.223.32    datacenter1 rack1       Up     Normal  191.1 MB
>> >> >  33.33%  56713727820156410577229101238628035242
>> >> > 10.86.229.24    datacenter1 rack1       Up     Normal  193.63 MB
>> >> > 33.33%  113427455640312821154458202477256070484
>> >> >
>> >> >
>> >> >
>> >> > SEED
>> >> > Pool Name                    Active   Pending      Completed
>> >> > ReadStage                         0         0              7
>> >> > RequestResponseStage              0         0        1479839
>> >> > MutationStage                     0         0        1479908
>> >> > ReadRepairStage                   0         0              0
>> >> > ReplicateOnWriteStage             0         0              0
>> >> > GossipStage                       0         0           5176
>> >> > AntiEntropyStage                  0         0              0
>> >> > MigrationStage                    0         0            128
>> >> > MemtablePostFlusher               0         0             28
>> >> > StreamStage                       0         0              0
>> >> > FlushWriter                       0         0             28
>> >> > MiscStage                         0         0              2
>> >> > FlushSorter                       0         0              0
>> >> > InternalResponseStage             0         0            363
>> >> > HintedHandoff                     0         0              0
>> >> > NODE1
>> >> > Pool Name                    Active   Pending      Completed
>> >> > ReadStage                         0         0              7
>> >> > RequestResponseStage              0         0        2960208
>> >> > MutationStage                     0         0        1480104
>> >> > ReadRepairStage                   0         0              0
>> >> > ReplicateOnWriteStage             0         0        1480104
>> >> > GossipStage                       0         0           4151
>> >> > AntiEntropyStage                  0         0              0
>> >> > MigrationStage                    0         0            128
>> >> > MemtablePostFlusher               0         0             31
>> >> > StreamStage                       0         0              8
>> >> > FlushWriter                       0         0             31
>> >> > MiscStage                         0         0              8
>> >> > FlushSorter                       0         0              0
>> >> > InternalResponseStage             0         0              1
>> >> > HintedHandoff                     0         0              0
>> >> > NODE2
>> >> > Pool Name                    Active   Pending      Completed
>> >> > ReadStage                         0         0              7
>> >> > RequestResponseStage              0         0              0
>> >> > MutationStage                     0         0        1481530
>> >> > ReadRepairStage                   0         0              0
>> >> > ReplicateOnWriteStage             0         0              0
>> >> > GossipStage                       0         0           4201
>> >> > AntiEntropyStage                  0         0              0
>> >> > MigrationStage                    0         0            128
>> >> > MemtablePostFlusher               0         0             31
>> >> > StreamStage                       0         0              6
>> >> > FlushWriter                       0         0             31
>> >> > MiscStage                         0         0              6
>> >> > FlushSorter                       0         0              0
>> >> > InternalResponseStage             0         0              1
>> >> > HintedHandoff                     0         0              0
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>>
>>
>>
>>  --
>>  Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
>

Mime
View raw message