cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shini gupta <gupta.sh...@gmail.com>
Subject Re: Repair with –pr stuck in between on Cassandra 3.11.1
Date Thu, 25 Jan 2018 14:30:17 GMT
Hi Paulo,
No we are not using JBOD.Just a bunch of disks.
Thanks

On Thu, 25 Jan 2018 at 5:44 PM, Paulo Motta <pauloricardomg@gmail.com>
wrote:

> Are you using JBOD? A thread dump (jstack <pid>) on the affected nodes
> would probably help troubleshoot this.
>
> 2018-01-25 6:45 GMT-02:00 shini gupta <gupta.shini@gmail.com>:
> > Hi,
> >
> >
> > We have upgraded the system from Cassandra 2.1.16 to 3.11.1. After about
> > 335M of data loading, repair  with –pr and –full option was triggered for
> > all keyspaces (7 keyspaces). Data size of main_k1 is ~170G.
> >
> >
> >
> > Please find the command executed for repair:
> >
> >
> >
> > …./cassandra/bin/nodetool -u <username> -<password> -p 7199 repair
-full
> –tr
> >
> >
> >
> > It was observed that the repair for the first keyspace i.e main_k1 got
> > finished in 2 hrs at 13:39 hrs. Even after about 1:30 hrs, there were
> just
> > warning messages about the orphaned sstables. The repair command for the
> > first keyspace completed successfully but then the nodetool repair
> command
> > got stuck and there was no output for remaining 6 keyspaces.
> >
> >
> >
> > Please find the system logs below:
> >
> >
> >
> > INFO  [Repair-Task-2] 2018-01-18 11:35:01,161 RepairRunnable.java:139 -
> > Starting repair command #1 (84dbf590-fc15-11e7-85c6-1594c4c73c8e),
> repairing
> > keyspace main_k1 with repair options (parallelism: parallel, primary
> range:
> > true, incremental: false, job threads: 1, ColumnFamilies: [],
> dataCenters:
> > [], hosts: [], # of ranges: 256, pull repair: false)
> >
> >
> >
> > INFO  [CompactionExecutor:39] 2018-01-18 13:39:58,174
> > RepairRunnable.java:343 - Repair command #1 finished in 2 hours 4
> minutes 57
> > seconds
> >
> >
> >
> > WARN  [ValidationExecutor:12] 2018-01-18 14:30:49,356
> > LeveledCompactionStrategy.java:273 - Live sstable
> >
> …../data/main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22184-big-Data.db
> > from level 3 is not on corresponding level in the leveled manifest. This
> is
> > not a problem per se, but may indicate an orphaned sstable due to a
> failed
> > compaction not cleaned up properly.
> >
> > ……………………………………………………………….
> >
> >
> >
> > WARN  [ValidationExecutor:26] 2018-01-18 15:03:53,598
> > LeveledCompactionStrategy.java:273 - Live sstable …../data
> > /main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22291-big-Data.db
> from
> > level 2 is not on corresponding level in the leveled manifest. This is
> not a
> > problem per se, but may indicate an orphaned sstable due to a failed
> > compaction not cleaned up properly.
> >
> > WARN  [ValidationExecutor:26] 2018-01-18 15:03:53,598
> > LeveledCompactionStrategy.java:273 - Live sstable …../data
> > /main_k1/table1-4b9c1fd0f4f411e7889bd9124bc6a6eb/mc-22216-big-Data.db
> from
> > level 3 is not on corresponding level in the leveled manifest. This is
> not a
> > problem per se, but may indicate an orphaned sstable due to a failed
> > compaction not cleaned up properly.
> >
> >
> >
> > Is anybody facing such issues with repair –pr for cassandra 3.11.1 ? Is
> this
> > behavior due to the warning messages reported in logs.
> >
> >
> >
> > Thanks.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
> --
-Shini Gupta

""Trusting in God won't make the mountain smaller,
But will make climbing easier.
Do not ask God for a lighter load
But ask Him for a stronger back... ""

Mime
View raw message