cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulo Motta <pauloricard...@gmail.com>
Subject Re: Repair with –pr stuck in between on Cassandra 3.11.1
Date Thu, 25 Jan 2018 12:13:45 GMT
Are you using JBOD? A thread dump (jstack <pid>) on the affected nodes
would probably help troubleshoot this.

2018-01-25 6:45 GMT-02:00 shini gupta <gupta.shini@gmail.com>:
> Hi,
>
>
> We have upgraded the system from Cassandra 2.1.16 to 3.11.1. After about
> 335M of data loading, repair  with –pr and –full option was triggered for
> all keyspaces (7 keyspaces). Data size of main_k1 is ~170G.
>
>
>
> Please find the command executed for repair:
>
>
>
> …./cassandra/bin/nodetool -u <username> -<password> -p 7199 repair -full
–tr
>
>
>
> It was observed that the repair for the first keyspace i.e main_k1 got
> finished in 2 hrs at 13:39 hrs. Even after about 1:30 hrs, there were just
> warning messages about the orphaned sstables. The repair command for the
> first keyspace completed successfully but then the nodetool repair command
> got stuck and there was no output for remaining 6 keyspaces.
>
>
>
> Please find the system logs below:
>
>
>
> INFO  [Repair-Task-2] 2018-01-18 11:35:01,161 RepairRunnable.java:139 -
> Starting repair command #1 (84dbf590-fc15-11e7-85c6-1594c4c73c8e), repairing
> keyspace main_k1 with repair options (parallelism: parallel, primary range:
> true, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters:
> [], hosts: [], # of ranges: 256, pull repair: false)
>
>
>
> INFO  [CompactionExecutor:39] 2018-01-18 13:39:58,174
> RepairRunnable.java:343 - Repair command #1 finished in 2 hours 4 minutes 57
> seconds
>
>
>
> WARN  [ValidationExecutor:12] 2018-01-18 14:30:49,356
> LeveledCompactionStrategy.java:273 - Live sstable
> …../data/main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22184-big-Data.db
> from level 3 is not on corresponding level in the leveled manifest. This is
> not a problem per se, but may indicate an orphaned sstable due to a failed
> compaction not cleaned up properly.
>
> ……………………………………………………………….
>
>
>
> WARN  [ValidationExecutor:26] 2018-01-18 15:03:53,598
> LeveledCompactionStrategy.java:273 - Live sstable …../data
> /main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22291-big-Data.db from
> level 2 is not on corresponding level in the leveled manifest. This is not a
> problem per se, but may indicate an orphaned sstable due to a failed
> compaction not cleaned up properly.
>
> WARN  [ValidationExecutor:26] 2018-01-18 15:03:53,598
> LeveledCompactionStrategy.java:273 - Live sstable …../data
> /main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22216-big-Data.db from
> level 3 is not on corresponding level in the leveled manifest. This is not a
> problem per se, but may indicate an orphaned sstable due to a failed
> compaction not cleaned up properly.
>
>
>
> Is anybody facing such issues with repair –pr for cassandra 3.11.1 ? Is this
> behavior due to the warning messages reported in logs.
>
>
>
> Thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message