db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Thalamati (JIRA)" <derby-...@db.apache.org>
Subject [jira] Commented: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
Date Tue, 18 Jul 2006 23:54:14 GMT
    [ http://issues.apache.org/jira/browse/DERBY-1156?page=comments#action_12422000 ] 
Suresh Thalamati commented on DERBY-1156:

   How to handle crash/recovery when switching to new encryption key ?

After re-encrypting all the containers in the database with a new 
encryption key. A final switch has to be made for the database to use 
the new encryption key to encrypt the transaction log and the data. 

Problem is how to perform the switch in such a way that if there is crash, 
system will work with the old encryption key or the new encryption key. 
Next boot of the database should not require both the keys to recovery
from a crash.

Log is encrypted, so recovery has to know the correct encryption 
key to decrypt the transaction log records.   Note that even the 
checkpoint log record in the transaction log is encrypted. 

I think  an algorithm using a flag in the log control file
(CHECK_POINT_WITH_NEW_KEY) tha is written on check-point and 
another flag (derby.storage.reencryptionLog) in service.properties 
to track that re-encryption in progress and also to remember the first 
log file that contains the log with the new encryption key.  

During the switch to new-encryption key , old copy of service.properties and 
the verify.key files saved , so that incase of crash, system will revert back
to the old version of these files. 

Incase of a crash during re-encryption ; On recovery  reading from 
the log that contains log records encrypted with new key 
including the commit log record of re-encryption is avoided by 
deleting the log file. Because the commit log record is written to 
the last log file, re-encryption transaction  will be rolled back bringing 
the database to state it was before re-encryption  started. 

Persistent Values/files :  
CHECK_POINT_WITH_NEW_KEY  to the log.ctrl file 
derby.storage.reencryptionLog property to the service.properties. 
service.properties.old. verify.key.old  to keep track of old version 
of the encryption properties. 

Following is the description of the  re-encryption of the database algorithm:

Re-encryption is performed after the recovery during the RawStore boot:

1) Perform  a check point, this will make sure old  transaction log 
   will not be read after the re-encryption (Logged with OLD KEY)

2) Begin Transaction   (Logged with OLD Key).

3) Re-encrypt all containers with the new key. (this action is 
   logged with OLD KEY).

5) Switch to a new Log File and set the encryption engine for the log  
   to be the one with the new encryption key. 

(After this point the transaction log will be encrypted with 
  the New Encryption Key).

6) update the service.properties with the property 
    derby.storage.reencryptionLog = (the last log file number)  
   (this property will help in tracking that re-encryption in progress
    and also to delete the last log file in-case of crash)

7) copy  service.properties after excluding derby.storage.reencryptionLog property 
   to service.properties.old  and and verify.key into verify.key.old

8) End Transaction   (logged with NEW KEY)

9) update the service.properties with new encryption key/password information. 

10) update the verify.key file with the new external key incase of external 
   encryption key. 

11) Perform a checkpoint and also mark a flag that checkpoint is done 
   with a new key in the log.ctrl file. CHECK_POINT_WITH_NEW_KEY = true. 


Now perform the cleanup:

performCleanup () 


12) Update the derby.storage.reencryptionLog = 0  ( re-encryption complete , 
   only cleanup remaining)

13) Cleanup copies of the container files encrypted with old encryption key.

14) Update the log.control file with CHECK_POINT_WITH_NEW_KEY = false.

15) Remove the service.properties.old and verify.key.old files. 

16) Remove derby.storage.reencryptionLog property from the service.properties


If all the steps above are successful,  database is ready to use the new
encryption key.


On Recovery :


   String renEncryptionLogProp = find(derby.storage.ReEncryptionLog );
   if( reEncryptioLogProp != null ) {
      reEncyrptionLog = Long.valueof(renEncryptionLog);
   } else {

       // This may not re-encryption recovery case or it crashed 
       // before the derby.storage.ReEncryptionLog was set.

       // database will continue to use the old key and recovery will 
       // rollback if there were any changed by re-encryption. No special 
       // handling required.       

  if ( reEncryptionLog > 0 )  {
        // renecryption was in progress 
         if (CHECK_POINT_WITH_NEW_KEY == true) 
           // crashed after the checkpoint with the new key , 
           // this is as good as re-encryption is complete.  
           1) call performCleanup();

           2) verify the user entered key against the new verify.key and
                proceed with recovery.
              // re-encryption was not complete, rollback and 
              // bring the database to the state it was before
              // re-encryption started. 

            1) delete(Blow up) the log file(derby.storage.ReEncryptionLog) 
               where log records with  new encryption key is written if 
               the file exists. (so that we don't see any log with new  key.

             2) revert to the the old versions of service.properties and
	         verify.key, this will also remove the
	         derby.storage.ReEncryptionLog property.

             3) verify the user entered old key/password using the
               verify.key/or the key in the service.properties. And then
               proceed with recovery.

             4) recovery will undo the partially completed re-encryption, 
                this will bring the containers back to versions of 
                the old encryption key.

  } else 
     if (reEncryptionLog == 0) 
        // crash occurred during cleanup; re-encryption was complete.
       call performcleanup();

If the above algorithm  works, Other minor issues:

1) instead of having service.properties.old file name the specific properties 
   as new or old and set them accordingly. 
2) instead of storing the log file to delete incase of crash in the
   as a property (derby.storage.ReEncryptionLog) in the
   service.properties. Store it in the log.ctrl file; There is only one LONG
   spare remaining , not sure if worth using for this case. 

I would really appreciate if some one can take a look at the above algorithm 
and verify if it really works or if there are any suggestions solve in
different way, that will be great too. 



> allow the encrypting of an existing unencrypted db and allow the re-encrypting of an
existing encrypted db
> ----------------------------------------------------------------------------------------------------------
>                 Key: DERBY-1156
>                 URL: http://issues.apache.org/jira/browse/DERBY-1156
>             Project: Derby
>          Issue Type: Improvement
>          Components: Store
>    Affects Versions:
>            Reporter: Mike Matrigali
>         Assigned To: Suresh Thalamati
>            Priority: Minor
>             Fix For:
>         Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, reencrypt_3.diff,
> encrypted database to be re-encrypted with a new password.
> Here are some ideas for an initial implementation.
> The easiest way to do this is to make sure we have exclusive access to the
> data and that no log is required in the new copy of the db.  I want to avoid
> the log as it also is encrypted.  Here is my VERY high level plan:
> 1) Force exclusive access by putting all the work in the low level store,
>    offline boot method.  We will do redo recovery as usual, but at the end
>    there will be an entry point to do the copy/encrypt operation.
> copy/encrypt process:
> 0) The request to encrypt/re-encrypt the db will be handled with a new set
>    of url flags passed into store at boot time.  The new flags will provide
>    the same inputs as the current encrypt flags.  So at high level the
>    request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags".
>    TODO - provide exact new flag syntax.
> 1) Open a transaction do all logged work to do the encryption.  All logging
>    will be done with existing encryption.
> 2) Copy and encrypt every db file in the database.  The target files will
>    be in the data directory.  There will be a new suffix to track the new
>    files, similar to the current process used for handling drop table in
>    a transaction consistent manner without logging the entire table to the log.
>    Entire encrypted destination file is guaranteed synced to disk before
>    transaction commits.  I don't think this part needs to be logged.
>    Files will be read from the cache using existing mechanism and written
>    directly into new encrypted files (new encrypted data does not end up in
>    the cache).
> 3) Switch encrypted files for old files.  Do this under a new log operation
>    so the process can be correctly rolled back if the encrypt db operation
>    transaction fails.  Rollback will do file at a time switches, no reading
>    of encrypted data is necessary.
> 4) log a "change encryption of db" log record, but do not update
>    system.properties with the change.
> 5) commit transaction.
> 6) update system.properties and sync changes.
> 7) TODO - need someway to handle crash between steps 5 and 6.
> 6) checkpoint all data, at this point guaranteed that there is no outstanding
>    transaction, so after checkpoint is done there is no need for the log.
> o there probably should be something that catches a request to encrypt to
>   whatever db was already encrypted with.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message