cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Roth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-12280) nodetool repair hangs
Date Tue, 16 Aug 2016 10:16:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-12280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15422548#comment-15422548
] 

Benjamin Roth commented on CASSANDRA-12280:
-------------------------------------------

More facts:

I dropped all material views for the affected base view. I just want to assert that the hangs
are not related to MVs. Result: No change. Problem persists.

The "hangs" can occur both when sending or receiving streams from the point of view of the
node that initiated the repair.
When there is a hanging stream I see large messages pending in the netstats of the stream
sender.

Example:

{{
Mode: NORMAL
Repair d8e84ba6-6398-11e6-9e03-55770f644c0a
    /10.23.71.101
        Receiving 12 files, 150755795 bytes total. Already received 1 files, 35803 bytes total
            visits/visits_in 35803/35803 bytes(100%) received from idx:0/10.23.71.101
        Sending 12 files, 148462450 bytes total. Already sent 6 files, 100752105 bytes total
            /var/lib/cassandra/data/visits/visits_in-7153d030326611e69c27e31e74028313/mb-374659-big-Data.db
20247100/20247100 bytes(100%) sent to idx:0/10.23.71.101
            /var/lib/cassandra/data/visits/visits_in-7153d030326611e69c27e31e74028313/mb-374669-big-Data.db
22245918/22245918 bytes(100%) sent to idx:0/10.23.71.101
            /var/lib/cassandra/data/visits/visits_in-7153d030326611e69c27e31e74028313/mb-368056-big-Data.db
19356082/19356082 bytes(100%) sent to idx:0/10.23.71.101
            /var/lib/cassandra/data/visits/visits_in-7153d030326611e69c27e31e74028313/mb-375121-big-Data.db
35844/35844 bytes(100%) sent to idx:0/10.23.71.101
            /var/lib/cassandra/data/visits/visits_in-7153d030326611e69c27e31e74028313/mb-363690-big-Data.db
2473776/22356954 bytes(11%) sent to idx:0/10.23.71.101
            /var/lib/cassandra/data/visits/visits_in-7153d030326611e69c27e31e74028313/mb-374955-big-Data.db
13885172/13885172 bytes(100%) sent to idx:0/10.23.71.101
            /var/lib/cassandra/data/visits/visits_in-7153d030326611e69c27e31e74028313/mb-374665-big-Data.db
22508213/22508213 bytes(100%) sent to idx:0/10.23.71.101
Read Repair Statistics:
Attempted: 2418
Mismatch (Blocking): 0
Mismatch (Background): 4
Pool Name                    Active   Pending      Completed   Dropped
Large messages                  n/a         3           5181         0
Small messages                  n/a         0         564875         0
Gossip messages                 n/a         0           8269         0
}}

> nodetool repair hangs
> ---------------------
>
>                 Key: CASSANDRA-12280
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12280
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Benjamin Roth
>
> nodetool repair hangs when repairing a keyspace, does not hang when repairting table/mv
by table/mv.
> Command executed (both variants make it hang):
> nodetool repair likes like dislike_by_source_mv like_by_contact_mv match_valid_mv like_out
dislike match match_by_contact_mv like_valid_mv like_out_by_source_mv
> OR
> nodetool repair likes
> Logs:
> https://gist.github.com/brstgt/bf8b20fa1942d29ab60926ede7340b75
> Nodetool output:
> https://gist.github.com/brstgt/3aa73662da4b0190630ac1aad6c90a6f
> Schema:
> https://gist.github.com/brstgt/3fd59e0166f86f8065085532e3638097



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message