incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Lee <James....@metaswitch.com>
Subject RE: Data not fully replicated with 2 nodes and replication factor 2
Date Wed, 19 Jun 2013 09:22:39 GMT
The test tool I am using catches any exceptions on the original writes and resubmits the write
request until it's successful (bailing out after 5 failures).  So for each key Cassandra has
reported a successful write.


Nodetool says the following - I'm guessing the pending hinted handoff is the interesting bit?

comet-mvs01:/dsc-cassandra-1.2.2# ./bin/nodetool tpstats
Pool Name                    Active   Pending      Completed   Blocked  All time blocked
ReadStage                         0         0          35445         0                 0
RequestResponseStage              0         0        1535171         0                 0
MutationStage                     0         0        3038941         0                 0
ReadRepairStage                   0         0           2695         0                 0
ReplicateOnWriteStage             0         0              0         0                 0
GossipStage                       0         0           2898         0                 0
AntiEntropyStage                  0         0              0         0                 0
MigrationStage                    0         0            245         0                 0
MemtablePostFlusher               0         0           1260         0                 0
FlushWriter                       0         0            633         0               212
MiscStage                         0         0              0         0                 0
commitlog_archiver                0         0              0         0                 0
InternalResponseStage             0         0              0         0                 0
HintedHandoff                     1         1              0         0                 0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                  0
BINARY                       0
READ                         0
MUTATION                 60427
_TRACE                       0
REQUEST_RESPONSE             0


Looking at the hints column family in the system keyspace, I see one row with a large number
of columns.  Presumably that along with the nodetool output above suggests there are hinted
handoffs pending?  How long should I expect these to remain for?

Ah, actually now that I re-run the command it seems that nodetool now reports that hint as
completed and there are no hints left in the system keyspace on either node.  I'm still seeing
failures to read the data I'm expecting though, as before.  Note that I've run this with a
smaller data set (2M rows, 1GB data total) for this latest test.

Thanks,
James


-----Original Message-----
From: Robert Coli [mailto:rcoli@eventbrite.com] 
Sent: 18 June 2013 19:45
To: user@cassandra.apache.org
Subject: Re: Data not fully replicated with 2 nodes and replication factor 2

On Tue, Jun 18, 2013 at 11:36 AM, Wei Zhu <wz1975@yahoo.com> wrote:
> Cassandra doesn't do async replication like HBase does.You can run 
> nodetool repair to insure the consistency.

While this answer is true, it is somewhat non-responsive to the OP.

If the OP didn't see timeout exception, the theoretical worst case is that he should have
hints stored for initially failed to replicate writes. His nodes should not be failing GC
with a total data size of 5gb on an 8gb heap, so those hints should deliver quite quickly.
After
30 minutes those hints should certainly be delivered.

@OP : do you see hints being stored? does nodetool tpstats indicate dropped messages?

=Rob

Mime
View raw message