Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Fri, 16 Dec 2016 13:01:00 +0000 (UTC)
From: "Jeremiah Jordan (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.13022760.1479884648000.538761.1481893260677@Atlassian.JIRA>
In-Reply-To: <JIRA.13022760.1479884648000@Atlassian.JIRA>
References: <JIRA.13022760.1479884648000@Atlassian.JIRA> <JIRA.13022760.1479884648566@arcas>
Subject: [jira] [Commented] (CASSANDRA-12947) Repair not replicating data
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 16 Dec 2016 13:01:02 -0000


    [ https://issues.apache.org/jira/browse/CASSANDRA-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15754356#comment-15754356 ] 

Jeremiah Jordan commented on CASSANDRA-12947:
---------------------------------------------

If you didn't run a full repair after running scrub, then this behavior is expected.

> Repair not replicating data
> ---------------------------
>
>                 Key: CASSANDRA-12947
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12947
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra 3.7
> Single DC
> 7 Nodes
> RF 3
> NetworkTopologyStrategy
> OS: Ubuntu
>            Reporter: Malte Pickhan
>
> We experienced strange behaviour of our C* cluster last week.
> In the logs we've seen multiple requests in the logs where we the values read from Cassandra actually have been 'null'. 
> When running the query on a node, we sometimes received the result and sometimes not.
> When setting the CL to LOCAL_QUORUM everything was fine. So far nothing unusual, probably the dataset wasn't replicated to one of the nodes.
> When turning on the tracing and running the query there was following intersting line:
> {quote}
> Initiating read-repair [SharedPool-Worker-2] | 2016-11-18 10:17:47.528000 |   $PUBLIC_IP |            126 | 127.0.0.1
>  Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(-5887526567589486157, 3130333031303338383436303937) (db1e86d507513ff12ba95f0eff984b60 vs d41d8cd98f00b204e9800998ecf8427e) [ReadRepairStage:1]
> {quote}
> This is probably related to CASSANDRA-12090?
> The interesting part is, after that we've run a 'nodetool repair -pr', after that the behaviour was still the same and the data randomly not available, depending on which node was hit.
> Only after running a 'nodetool repair -pr -full' the issue was gone.
> Did we miss something here? The point that's bothering me is that the dataset was not replicated.
> Worth to note is probably that some weeks ago we've hit the bug CASSANDRA-12694 and fixed it by scrubbing some tables.
> Any hints/help are appreciated.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)