Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of James.Lee@metaswitch.com
 designates 192.91.191.38 as permitted sender)
From: James Lee <James.Lee@metaswitch.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: RE: Data not fully replicated with 2 nodes and replication factor 2
Thread-Topic: Data not fully replicated with 2 nodes and replication factor 2
Thread-Index: AQHObFKzVbm8IwNvSQSkSX38R8PWVpk7vpaAgAEBnYA=
Date: Wed, 19 Jun 2013 09:22:39 +0000
Message-ID: <884F6606C2AA344EB8D08020872FE8AEC19AE7B3@ENFICSMBX1.datcon.co.uk>
References: <1371580569.56290.GenericBBA@web160905.mail.bf1.yahoo.com>
 <CAEDUwd1E_zqLh7=tTCe2cE5Y6y+kkzJ8gq1iPhZm=KYBN8oMbw@mail.gmail.com>
In-Reply-To: 
 <CAEDUwd1E_zqLh7=tTCe2cE5Y6y+kkzJ8gq1iPhZm=KYBN8oMbw@mail.gmail.com>
Accept-Language: en-GB, en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

The test tool I am using catches any exceptions on the original writes and =
resubmits the write request until it's successful (bailing out after 5 fail=
ures).  So for each key Cassandra has reported a successful write.


Nodetool says the following - I'm guessing the pending hinted handoff is th=
e interesting bit?

comet-mvs01:/dsc-cassandra-1.2.2# ./bin/nodetool tpstats
Pool Name                    Active   Pending      Completed   Blocked  All=
 time blocked
ReadStage                         0         0          35445         0     =
            0
RequestResponseStage              0         0        1535171         0     =
            0
MutationStage                     0         0        3038941         0     =
            0
ReadRepairStage                   0         0           2695         0     =
            0
ReplicateOnWriteStage             0         0              0         0     =
            0
GossipStage                       0         0           2898         0     =
            0
AntiEntropyStage                  0         0              0         0     =
            0
MigrationStage                    0         0            245         0     =
            0
MemtablePostFlusher               0         0           1260         0     =
            0
FlushWriter                       0         0            633         0     =
          212
MiscStage                         0         0              0         0     =
            0
commitlog_archiver                0         0              0         0     =
            0
InternalResponseStage             0         0              0         0     =
            0
HintedHandoff                     1         1              0         0     =
            0

Message type           Dropped
RANGE_SLICE                  0
READ_REPAIR                  0
BINARY                       0
READ                         0
MUTATION                 60427
_TRACE                       0
REQUEST_RESPONSE             0


Looking at the hints column family in the system keyspace, I see one row wi=
th a large number of columns.  Presumably that along with the nodetool outp=
ut above suggests there are hinted handoffs pending?  How long should I exp=
ect these to remain for?

Ah, actually now that I re-run the command it seems that nodetool now repor=
ts that hint as completed and there are no hints left in the system keyspac=
e on either node.  I'm still seeing failures to read the data I'm expecting=
 though, as before.  Note that I've run this with a smaller data set (2M ro=
ws, 1GB data total) for this latest test.

Thanks,
James


-----Original Message-----
From: Robert Coli [mailto:rcoli@eventbrite.com]=20
Sent: 18 June 2013 19:45
To: user@cassandra.apache.org
Subject: Re: Data not fully replicated with 2 nodes and replication factor =
2

On Tue, Jun 18, 2013 at 11:36 AM, Wei Zhu <wz1975@yahoo.com> wrote:
> Cassandra doesn't do async replication like HBase does.You can run=20
> nodetool repair to insure the consistency.

While this answer is true, it is somewhat non-responsive to the OP.

If the OP didn't see timeout exception, the theoretical worst case is that =
he should have hints stored for initially failed to replicate writes. His n=
odes should not be failing GC with a total data size of 5gb on an 8gb heap,=
 so those hints should deliver quite quickly. After
30 minutes those hints should certainly be delivered.

@OP : do you see hints being stored? does nodetool tpstats indicate dropped=
 messages?

=3DRob