incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <mdione....@orange.com>
Subject RE: RE Restore snapshot
Date Mon, 13 Aug 2012 09:31:04 GMT
> DeĀ : Sylvain Lebresne [mailto:sylvain@datastax.com]
> > 2) copy the snapshot sstable in the right place and call the JMX
> > method
> > loadNewSSTables() (in the column family MBean, which mean you need to
> > do that per-CF).
> 
> 	How does this affect the contents of the CommitLogs? I mean, I
> imagine using this for rollback'ing errors in the app side, like
> some/many/all objects deleted. How will updates/deletes saved in
> CommitLogs impact the data restored this way? Also: while I'm restoring
> a KS of CF, is it possible to cut write requests but not reads? I hope
> I'm explaining myself clearly...

  In particular, I'm thinking on a restore like this:

* the app does something stupid.
* (if possible) I stop writes to the KS or CF.
* remove the db files with the wrong data.
* I put the db files from the restore.
* nodetool refresh
* profit!

  But on Friday I did it slightly different (too tired, I was confused), 
and only did the refresh on one node and then a repair on the same, and 
it failed like this:

* state just after the app did something stupid:

root@pnscassandra04:~# tree /var/opt/hosting/db/cassandra/data/one_cf/cf_1/
/var/opt/hosting/db/cassandra/data/one_cf/cf_1/
|-- one_cf-cf_1-hd-13-CompressionInfo.db
|-- one_cf-cf_1-hd-13-Data.db
|-- one_cf-cf_1-hd-13-Filter.db
|-- one_cf-cf_1-hd-13-Index.db
|-- one_cf-cf_1-hd-13-Statistics.db
|-- one_cf-cf_1-hd-14-CompressionInfo.db
|-- one_cf-cf_1-hd-14-Data.db
|-- one_cf-cf_1-hd-14-Filter.db
|-- one_cf-cf_1-hd-14-Index.db
|-- one_cf-cf_1-hd-14-Statistics.db
|-- one_cf-cf_1-hd-15-CompressionInfo.db
|-- one_cf-cf_1-hd-15-Data.db
|-- one_cf-cf_1-hd-15-Filter.db
|-- one_cf-cf_1-hd-15-Index.db
|-- one_cf-cf_1-hd-15-Statistics.db
`-- snapshots
    `-- 1344609231532
        |-- one_cf-cf_1-hd-10-CompressionInfo.db
        |-- one_cf-cf_1-hd-10-Data.db
        |-- one_cf-cf_1-hd-10-Filter.db
        |-- one_cf-cf_1-hd-10-Index.db
        |-- one_cf-cf_1-hd-10-Statistics.db
        |-- one_cf-cf_1-hd-11-CompressionInfo.db
        |-- one_cf-cf_1-hd-11-Data.db
        |-- one_cf-cf_1-hd-11-Filter.db
        |-- one_cf-cf_1-hd-11-Index.db
        |-- one_cf-cf_1-hd-11-Statistics.db
        |-- one_cf-cf_1-hd-12-CompressionInfo.db
        |-- one_cf-cf_1-hd-12-Data.db
        |-- one_cf-cf_1-hd-12-Filter.db
        |-- one_cf-cf_1-hd-12-Index.db
        |-- one_cf-cf_1-hd-12-Statistics.db
        |-- one_cf-cf_1-hd-9-CompressionInfo.db
        |-- one_cf-cf_1-hd-9-Data.db
        |-- one_cf-cf_1-hd-9-Filter.db
        |-- one_cf-cf_1-hd-9-Index.db
        `-- one_cf-cf_1-hd-9-Statistics.db

* I remove the 'wrong' databases

root@pnscassandra04:~# rm -v /var/opt/hosting/db/cassandra/data/one_cf/cf_1/*.db
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-CompressionInfo.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Data.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Filter.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Index.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Statistics.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-CompressionInfo.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Data.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Filter.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Index.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-14-Statistics.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-CompressionInfo.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Data.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Filter.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Index.db'
removed `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-15-Statistics.db'

* restore

root@pnscassandra04:~# for i in /var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/*;
do 
    ln -v $i /var/opt/hosting/db/cassandra/data/one_cf/cf_1/; 
done
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-CompressionInfo.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-CompressionInfo.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Data.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Data.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Filter.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Filter.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Index.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Index.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10-Statistics.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-10-Statistics.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-CompressionInfo.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-CompressionInfo.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-Data.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-Data.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-Filter.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-Filter.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-Index.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-Index.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11-Statistics.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-11-Statistics.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-CompressionInfo.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-CompressionInfo.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-Data.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-Data.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-Filter.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-Filter.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-Index.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-Index.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12-Statistics.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-12-Statistics.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-CompressionInfo.db' =>
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-CompressionInfo.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-Data.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-Data.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-Filter.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-Filter.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-Index.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-Index.db'
`/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9-Statistics.db' => `/var/opt/hosting/db/cassandra/data/one_cf/cf_1/snapshots/1344609231532/one_cf-cf_1-hd-9-Statistics.db'

* and refresh

root@pnscassandra04:~# nodetool refresh one_cf cf_1

INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,426 ColumnFamilyStore.java
(line 460) Loading new SSTables for one_cf/cf_1...
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,432 ColumnFamilyStore.java
(line 488) Renaming new SSTable /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-9
to /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-16
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,432 SSTableReader.java (line
170) Opening /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-16 (2263898 bytes)
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,488 ColumnFamilyStore.java
(line 488) Renaming new SSTable /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-11
to /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-17
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,489 SSTableReader.java (line
170) Opening /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-17 (2124847 bytes)
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,517 ColumnFamilyStore.java
(line 488) Renaming new SSTable /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-10
to /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-18
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,517 SSTableReader.java (line
170) Opening /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-18 (2526508 bytes)
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,551 ColumnFamilyStore.java
(line 488) Renaming new SSTable /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-12
to /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-19
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,551 SSTableReader.java (line
170) Opening /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-19 (146340 bytes)
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,555 ColumnFamilyStore.java
(line 510) Loading new SSTables and building secondary indexes for one_cf/cf_1: [SSTableReader(path='/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-17-Data.db'),
SSTableReader(path='/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-19-Data.db'),
SSTableReader(path='/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-16-Data.db'),
SSTableReader(path='/var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-18-Data.db')]
INFO [RMI TCP Connection(54)-10.234.169.244] 2012-08-10 17:00:54,555 ColumnFamilyStore.java
(line 526) Done loading load new SSTables for one_cf/cf_1

* and here's the error at repair time:

root@pnscassandra04:~# nodetool repair one_cf cf_1

INFO [RMI TCP Connection(56)-10.234.169.244] 2012-08-10 17:03:12,354 StorageService.java (line
1925) Starting repair command #3, repairing 5 ranges.
INFO [AntiEntropySessions:5] 2012-08-10 17:03:12,358 AntiEntropyService.java (line 666) [repair
#811be330-e2fc-11e1-0000-b64817724cbd] new session: will sync pnscassandra04.vdev.bas.s1.p.fti.net/10.234.169.244,
/10.234.169.245, /10.234.169.241, /10.234.169.242, /10.234.169.243 on range (34028236692093846346337460743176821145,68056473384187692692674921486353642291]
for one_cf.[cf_1]
INFO [AntiEntropySessions:5] 2012-08-10 17:03:12,359 AntiEntropyService.java (line 871) [repair
#811be330-e2fc-11e1-0000-b64817724cbd] requesting merkle trees for cf_1 (to [/10.234.169.245,
/10.234.169.241, /10.234.169.242, /10.234.169.243, pnscassandra04.vdev.bas.s1.p.fti.net/10.234.169.244])
ERROR [ValidationExecutor:2] 2012-08-10 17:03:12,361 AbstractCassandraDaemon.java (line 134)
Exception in thread Thread[ValidationExecutor:2,1,main]
java.io.IOError: java.io.FileNotFoundException: /var/opt/hosting/db/cassandra/data/one_cf/cf_1/one_cf-cf_1-hd-13-Data.db
(No such file or directory)
[...]

  I understand that this procedure (refresh+repair in one node) makes absolutely 
no sense, but the error leads me to think that after refresh, the 'wrong' databases
are still taken in account. Is it that the whole procedure is bad from the start and 
I should use SSTableLoader instead?

--
Marcos Dione
SysAdmin
Astek Sud-Est
pour FT/TGPF/OPF/PORTAIL/DOP/HEBEX @ Marco Polo
04 97 12 62 45 - mdione.ext@orange.com

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees
et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par
erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant
susceptibles d'alteration,
France Telecom - Orange decline toute responsabilite si ce message a ete altere, deforme ou
falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may
be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message
and its attachments.
As emails may be altered, France Telecom - Orange is not liable for messages that have been
modified, changed or falsified.
Thank you.


Mime
View raw message