Thanks! This would make sense but I'm not sure it maps to what I'm seeing.
Specifically if I start processes that write to each of the three nodes, so
I know there is a constant stream of writes to each node, instead of seeing
replicate on write on all nodes, I see it only on a single node.
It appears that all increments are being routed to a single node which then
has to do all replication. However I don't have a good way to see the write
flow
Connection made to node: XXX
Write sent to node: XXX
Replicas sent to nodes: XXX, XXX
If I could see that I'd have a lot more confidence about what's going on.
Ian
On Thu, Aug 11, 2011 at 6:38 PM, aaron morton <aaron@thelastpickle.com>wrote:
> I've not checked the code but from memory when the nodes are ordered in
> proximity to the coordinator the local node is always first if it's in the
> replica set. So with RF=3 and N=3 the closest node is always the local one.
>
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 12 Aug 2011, at 10:10, Ian Danforth wrote:
>
> I'm writing at QUORUM though and (pardon me for being dense) it would seem
> that the first replica shouldn't always be on the same server if I'm using
> RP.
>
> I very much appreciate your time, I'm sure there is something basic that
> just hasn't clicked.
>
> Ian
>
> On Thu, Aug 11, 2011 at 2:56 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>
>> https://issues.apache.org/jira/browse/CASSANDRA-2889
>>
>> On Thu, Aug 11, 2011 at 2:55 PM, Ian Danforth <idanforth@numenta.com>
>> wrote:
>> > I don't think so, perhaps more succinctly, why would ROW actions only be
>> > performed on a single node?
>> > Ian
>> >
>> > On Wed, Aug 10, 2011 at 8:12 PM, Jonathan Ellis <jbellis@gmail.com>
>> wrote:
>> >>
>> >> are you i/o bound? I believe counter ROW needs to perform a read of
>> >> the old value.
>> >>
>> >> On Wed, Aug 10, 2011 at 7:17 PM, Ian Danforth <idanforth@numenta.com>
>> >> wrote:
>> >> > Hello all!
>> >> > Thanks for taking the time to read this, I'm new to Cassandra and
>> really
>> >> > want to get it working :)
>> >> > Below you'll find the output from tpstats on the three nodes in my
>> .8.1
>> >> > cluster. As you'll note from ring (also below), my cluster appears
>> >> > balanced.
>> >> > However, please note that NODE1 is the only node that has completed
>> any
>> >> > ReplicateOnWriteStage actions. I've been having a lot of trouble with
>> >> > RoW
>> >> > and seen a single node in my various test clusters get hugely backed
>> up
>> >> > on
>> >> > this stage to the point where it slows all writes to a crawl.
>> >> > I'm using counters in a column family defined like this:
>> >> > ==== KEYSPACE/CF ====
>> >> > create keyspace Keyspace1
>> >> > with strategy_options=[{replication_factor:3}]
>> >> > and placement_strategy =
>> >> > 'org.apache.cassandra.locator.SimpleStrategy';
>> >> > use Keyspace1;
>> >> > create column family TwitterTest
>> >> > with default_validation_class = CounterColumnType
>> >> > and replicate_on_write = true;
>> >> > ====
>> >> >
>> >> > With any given batch_insert() I'm incrementing ~17,000 counters. I've
>> >> > backed
>> >> > off a bit and started using ConsistencyLevel.QUORUM, as it was
>> suggested
>> >> > elsewhere that ANY or ONE could overwhelm the system.
>> >> >
>> >> > I hope I'm just doing something stupid and one you experienced folks
>> can
>> >> > point out my error.
>> >> > Thanks in advance!
>> >> > Ian
>> >> > ==== MORE DATA ====
>> >> > Address DC Rack Status State Load
>> >> > Owns
>> >> > Token
>> >> >
>> >> > 113427455640312821154458202477256070484
>> >> > 10.87.9.57 datacenter1 rack1 Up Normal 193.86 MB
>> >> > 33.33% 0
>> >> > 10.86.223.32 datacenter1 rack1 Up Normal 191.1 MB
>> >> > 33.33% 56713727820156410577229101238628035242
>> >> > 10.86.229.24 datacenter1 rack1 Up Normal 193.63 MB
>> >> > 33.33% 113427455640312821154458202477256070484
>> >> >
>> >> >
>> >> >
>> >> > SEED
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 1479839
>> >> > MutationStage 0 0 1479908
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 0
>> >> > GossipStage 0 0 5176
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 28
>> >> > StreamStage 0 0 0
>> >> > FlushWriter 0 0 28
>> >> > MiscStage 0 0 2
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 363
>> >> > HintedHandoff 0 0 0
>> >> > NODE1
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 2960208
>> >> > MutationStage 0 0 1480104
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 1480104
>> >> > GossipStage 0 0 4151
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 31
>> >> > StreamStage 0 0 8
>> >> > FlushWriter 0 0 31
>> >> > MiscStage 0 0 8
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 1
>> >> > HintedHandoff 0 0 0
>> >> > NODE2
>> >> > Pool Name Active Pending Completed
>> >> > ReadStage 0 0 7
>> >> > RequestResponseStage 0 0 0
>> >> > MutationStage 0 0 1481530
>> >> > ReadRepairStage 0 0 0
>> >> > ReplicateOnWriteStage 0 0 0
>> >> > GossipStage 0 0 4201
>> >> > AntiEntropyStage 0 0 0
>> >> > MigrationStage 0 0 128
>> >> > MemtablePostFlusher 0 0 31
>> >> > StreamStage 0 0 6
>> >> > FlushWriter 0 0 31
>> >> > MiscStage 0 0 6
>> >> > FlushSorter 0 0 0
>> >> > InternalResponseStage 0 0 1
>> >> > HintedHandoff 0 0 0
>> >>
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of DataStax, the source for professional Cassandra support
>> >> http://www.datastax.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
>
|