cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Bromhead (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9291) Too many tombstones in schema_columns from creating too many CFs
Date Thu, 06 Aug 2015 21:36:06 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14660844#comment-14660844
] 

Ben Bromhead commented on CASSANDRA-9291:
-----------------------------------------

It worked for us but took a little bit of doing it a few times on the impacted nodes until
we got them to stream from the right nodes. You need to have more good nodes than broken ones
in this case (or at least one).

resetLocalSchema() in the MigrationManager picks the first node from Gossiper.instance.getLiveMembers(),
so if you have enough (or at least one) good node in your cluster you can eventually get to
a state of normality.

I think you can get the liveMembers list via JMX (correct me if I'm wrong here) to help figure
out which node it will use, otherwise you can roll through the whole cluster excluding the
known good ones. 

> Too many tombstones in schema_columns from creating too many CFs
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-9291
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9291
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Production Cluster with 2 DCs of 3 nodes each and 1 DC of 7 nodes,
running on dedicated Xeon hexacore, 96GB ram, RAID for Data and SSF for commitlog, running
Debian 7 (with Java 1.7.0_76-b13 64-Bit, 8GB and 16GB of heap tested).
> Dev Cluster with 1 DC with 3 nodes and 1 DC with 1 node, running on virtualized env.,
Ubuntu 12.04.5 (with Java 1.7.0_72-b14 64-Bit 1GB, 4GB heap) 
>            Reporter: Luis Correia
>            Priority: Blocker
>         Attachments: after_schema.txt, before_schema.txt, schemas500.cql
>
>
> When creating lots of columnfamilies (about 200) the system.schema_columns gets filled
with tombstones and therefore prevents clients using the binary protocol of connecting.
> Clients already connected continue normal operation (reading and inserting).
> Log messages are:
> For the first tries (sorry for the lack of precision):
> bq. ERROR [main] 2015-04-22 00:01:38,527 SliceQueryFilter.java (line 200) Scanned over
100000 tombstones in system.schema_columns; query aborted (see tombstone_failure_threshold)
> For each client that tries to connect but fails with timeout:
>  bq. WARN [ReadStage:35] 2015-04-27 15:40:10,600 SliceQueryFilter.java (line 231) Read
395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147283441 columns was requested, slices=[-]
> bq. WARN [ReadStage:40] 2015-04-27 15:40:10,609 SliceQueryFilter.java (line 231) Read
395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147283441 columns was requested, slices=[-]
> bq. WARN [ReadStage:61] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read
395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147283441 columns was requested, slices=[-]
> bq. WARN [ReadStage:51] 2015-04-27 15:40:10,670 SliceQueryFilter.java (line 231) Read
395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147283441 columns was requested, slices=[-]
> bq. WARN [ReadStage:55] 2015-04-27 15:40:10,675 SliceQueryFilter.java (line 231) Read
395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147283441 columns was requested, slices=[-]
> bq. WARN [ReadStage:35] 2015-04-27 15:40:10,707 SliceQueryFilter.java (line 231) Read
1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147282894 columns was requested, slices=[-]
> bq. WARN [ReadStage:40] 2015-04-27 15:40:10,708 SliceQueryFilter.java (line 231) Read
1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147282894 columns was requested, slices=[-]
> bq. WARN [ReadStage:43] 2015-04-27 15:40:10,715 SliceQueryFilter.java (line 231) Read
395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147283441 columns was requested, slices=[-]
> bq. WARN [ReadStage:51] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read
1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147282894 columns was requested, slices=[-]
> bq. WARN [ReadStage:61] 2015-04-27 15:40:10,736 SliceQueryFilter.java (line 231) Read
1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147282894 columns was requested, slices=[-]
> bq. WARN [ReadStage:35] 2015-04-27 15:40:10,750 SliceQueryFilter.java (line 231) Read
864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147281748 columns was requested, slices=[-]
> bq. WARN [ReadStage:40] 2015-04-27 15:40:10,751 SliceQueryFilter.java (line 231) Read
864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147281748 columns was requested, slices=[-]
> bq. WARN [ReadStage:55] 2015-04-27 15:40:10,759 SliceQueryFilter.java (line 231) Read
1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147282894 columns was requested, slices=[-]
> bq. WARN [ReadStage:51] 2015-04-27 15:40:10,821 SliceQueryFilter.java (line 231) Read
864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147281748 columns was requested, slices=[-]
> bq. WARN [ReadStage:61] 2015-04-27 15:40:10,822 SliceQueryFilter.java (line 231) Read
864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147281748 columns was requested, slices=[-]
> bq. WARN [ReadStage:43] 2015-04-27 15:40:10,827 SliceQueryFilter.java (line 231) Read
1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147282894 columns was requested, slices=[-]
> bq. WARN [ReadStage:55] 2015-04-27 15:40:10,838 SliceQueryFilter.java (line 231) Read
864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147281748 columns was requested, slices=[-]
> bq. WARN [ReadStage:62] 2015-04-27 15:40:10,846 SliceQueryFilter.java (line 231) Read
395 live and 1217 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147283441 columns was requested, slices=[-]
> bq. WARN [ReadStage:43] 2015-04-27 15:40:10,862 SliceQueryFilter.java (line 231) Read
864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147281748 columns was requested, slices=[-]
> bq. WARN [ReadStage:62] 2015-04-27 15:40:10,898 SliceQueryFilter.java (line 231) Read
1146 live and 3534 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147282894 columns was requested, slices=[-]
> bq. WARN [ReadStage:62] 2015-04-27 15:40:10,970 SliceQueryFilter.java (line 231) Read
864 live and 2664 tombstoned cells in system.schema_columns (see tombstone_warn_threshold).
2147281748 columns was requested, slices=[-]
> This happens independently of values in:
> tombstone_warn_threshold, tombstone_failure_threshold
> Thrift doesn't seem to be vulnerable to this, as cqlsh connects every time.
> Binary protocol tested with Java (and Clojure), Python, C++ all fail to connect.
> Included in attachment are a cql script to replicate (with 500 CFs) and a sstablemetadata
output of a system.schema_columns sstable before (clean cluster) and after importing the schema.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message