cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <j...@basetechnology.com>
Subject Re: All writes fail with ONE consistency level when adding second node to cluster?
Date Wed, 23 Jul 2014 15:40:32 GMT
Besides the obviously confusing error message, this particular case could simply be that the
hash value of the primary key belonged to the other node that wasn’t up, so even though
one node was up, it didn’t own that particular hash value or token, so CL=ONE could not
succeed.

What was RF set to for this two node cluster?

-- Jack Krupansky

From: Andrew 
Sent: Wednesday, July 23, 2014 1:02 AM
To: graham sanderson ; user@cassandra.apache.org 
Cc: Kevin Burton 
Subject: Re: All writes fail with ONE consistency level when adding second node to cluster?

I looked into this; ONE means it must be written to one replica—i.e., a node the data is
supposed to be written to.  ANY means a hinted handoff will “count”.  So as long as it
writes to any node on the cluster—even one that it’s not supposed to be on—it will be
a success.  Good to know.

Andrew


On July 22, 2014 at 8:13:57 PM, graham sanderson (graham@vast.com) wrote:

  Incorrect, ONE does not refer to the number of “other" nodes, it just refers to the number
of nodes. so ONE under normal circumstances would only require one node to acknowledge the
write. 

  The confusing error message you are getting is related to https://issues.apache.org/jira/browse/CASSANDRA-833…
Kevin you are correct in that normally that error message would make no sense.

  I don’t have much experience adding/removing nodes, but I think what is happening is that
your new node is in the middle of taken over ownership of a token range - while that happens
C* is trying to write to both the old owner (your original node), AND (hence the 2 not 1 in
the error message) the new owner (the new node) so that once the bootstrapping of the new
node is complete, it is immediately safe to delete the (no longer owned data) from the old
node. For whatever reason the write to the new node is timing out, causing the exception,
and the error message is exposing the “2” which happens to be how many C* thinks it is
waiting for at the time (i.e. how many it should be waiting for based on the consistency level
(1) plus this extra node).


  On Jul 22, 2014, at 9:46 PM, Andrew <redmumba@gmail.com> wrote:


    ONE means write to one replica (in addition to the original).  If you want to write to
any of them, use ANY.  Is that the right understanding?

    http://www.datastax.com/docs/1.0/dml/data_consistency

    Andrew


    On July 22, 2014 at 7:43:43 PM, Kevin Burton (burton@spinn3r.com) wrote:

      I'm super confused by this.. and disturbed that this was my failure scenario :-( 


      I had one cassandra node for the alpha of my app… and now we're moving into beta…
which means three replicas.


      So I added the second node… but my app immediately broke with:


      ""Cassandra timeout during write query at consistency ONE (2 replica were required but
only 1 acknowledged the write)""


      … but that makes no sense… if I'm at ONE and I have one acknowledged write, why
does it matter that the second one hasn't ack'd yet…


      ?


      --


      Founder/CEO Spinn3r.com

      Location: San Francisco, CA

      blog: http://burtonator.wordpress.com
      … or check out my Google+ profile 



------------------------------------------------------------------------------

Mime
View raw message