db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: [jira] Commented: (DERBY-1156) allow the encrypting of an existing unencrypted db and allow the re-encrypting of an existing encrypted db
Date Thu, 20 Jul 2006 19:54:05 GMT
My reading of the proposal is that it should work.  I've thought about 
this edge case a lot and have not come up with a better approach.  This
is really a very small window, so definitely pick anything that is
simpler (to understand/and code) over other solutions.  The description
is sort of high level now, if you want more feedback before submitting
a patch once you get deeper in the code and can post more detail to
the algorithm I will respond to whatever is posted.

I was sort of expecting old/new named properties in service.properties
vs. different versions of the file.  Again if different files are easier
that is fine with me.

As usual with these tiny crash point recovery cases, reproducible 
testing is a pain.  I am ok with using the "sane" only debug flag
method used in few other places in the store code.  The obvious
test points seem to be:
1) right before commit of reencryption (everything should be backed out 
and db only accessible with old key)
2) right after commit but before service.properties is updated on disk
    with new encryption info.
maybe others - basically make sure to test each new path you describe in
the recovery code described below.

Another good case might be doing the whole thing twice on the same db to
make sure we don't leave stuff around from the first time around.  ie.
encrypt unencrypted-> then reencrypt with a new key (with various 
crashes the second time).


Suresh Thalamati (JIRA) wrote:
>     [ http://issues.apache.org/jira/browse/DERBY-1156?page=comments#action_12422000 ]

> Suresh Thalamati commented on DERBY-1156:
> -----------------------------------------
>    How to handle crash/recovery when switching to new encryption key ?
> After re-encrypting all the containers in the database with a new 
> encryption key. A final switch has to be made for the database to use 
> the new encryption key to encrypt the transaction log and the data. 
> Problem is how to perform the switch in such a way that if there is crash, 
> system will work with the old encryption key or the new encryption key. 
> Next boot of the database should not require both the keys to recovery
> from a crash.
> Log is encrypted, so recovery has to know the correct encryption 
> key to decrypt the transaction log records.   Note that even the 
> checkpoint log record in the transaction log is encrypted. 
> I think  an algorithm using a flag in the log control file
> (CHECK_POINT_WITH_NEW_KEY) tha is written on check-point and 
> another flag (derby.storage.reencryptionLog) in service.properties 
> to track that re-encryption in progress and also to remember the first 
> log file that contains the log with the new encryption key.  
> During the switch to new-encryption key , old copy of service.properties and 
> the verify.key files saved , so that incase of crash, system will revert back
> to the old version of these files. 
> Incase of a crash during re-encryption ; On recovery  reading from 
> the log that contains log records encrypted with new key 
> including the commit log record of re-encryption is avoided by 
> deleting the log file. Because the commit log record is written to 
> the last log file, re-encryption transaction  will be rolled back bringing 
> the database to state it was before re-encryption  started. 
> Persistent Values/files :  
> CHECK_POINT_WITH_NEW_KEY  to the log.ctrl file 
> derby.storage.reencryptionLog property to the service.properties. 
> service.properties.old. verify.key.old  to keep track of old version 
> of the encryption properties. 
> Following is the description of the  re-encryption of the database algorithm:
> Re-encryption is performed after the recovery during the RawStore boot:
> 1) Perform  a check point, this will make sure old  transaction log 
>    will not be read after the re-encryption (Logged with OLD KEY)
> 2) Begin Transaction   (Logged with OLD Key).
> 3) Re-encrypt all containers with the new key. (this action is 
>    logged with OLD KEY).
> 5) Switch to a new Log File and set the encryption engine for the log  
>    to be the one with the new encryption key. 
> (After this point the transaction log will be encrypted with 
>   the New Encryption Key).
> 6) update the service.properties with the property 
>     derby.storage.reencryptionLog = (the last log file number)  
>    (this property will help in tracking that re-encryption in progress
>     and also to delete the last log file in-case of crash)
> 7) copy  service.properties after excluding derby.storage.reencryptionLog property 
>    to service.properties.old  and and verify.key into verify.key.old
> 8) End Transaction   (logged with NEW KEY)
> 9) update the service.properties with new encryption key/password information. 
> 10) update the verify.key file with the new external key incase of external 
>    encryption key. 
> 11) Perform a checkpoint and also mark a flag that checkpoint is done 
>    with a new key in the log.ctrl file. CHECK_POINT_WITH_NEW_KEY = true. 
> Now perform the cleanup:
> performCleanup () 
> {
> 12) Update the derby.storage.reencryptionLog = 0  ( re-encryption complete , 
>    only cleanup remaining)
> 13) Cleanup copies of the container files encrypted with old encryption key.
> 14) Update the log.control file with CHECK_POINT_WITH_NEW_KEY = false.
> 15) Remove the service.properties.old and verify.key.old files. 
> 16) Remove derby.storage.reencryptionLog property from the service.properties
>     file. 
> }
> If all the steps above are successful,  database is ready to use the new
> encryption key.
> On Recovery :
> 1) 
>    String renEncryptionLogProp = find(derby.storage.ReEncryptionLog );
>    if( reEncryptioLogProp != null ) {
>       reEncyrptionLog = Long.valueof(renEncryptionLog);
>    } else {
>        // This may not re-encryption recovery case or it crashed 
>        // before the derby.storage.ReEncryptionLog was set.
>        // database will continue to use the old key and recovery will 
>        // rollback if there were any changed by re-encryption. No special 
>        // handling required.       
>    }
>   if ( reEncryptionLog > 0 )  {
>         // renecryption was in progress 
>          if (CHECK_POINT_WITH_NEW_KEY == true) 
>          {
>            // crashed after the checkpoint with the new key , 
>            // this is as good as re-encryption is complete.  
>            1) call performCleanup();
>            2) verify the user entered key against the new verify.key and
>                 proceed with recovery.
>           }else 
>           {
>               // re-encryption was not complete, rollback and 
>               // bring the database to the state it was before
>               // re-encryption started. 
>             1) delete(Blow up) the log file(derby.storage.ReEncryptionLog) 
>                where log records with  new encryption key is written if 
>                the file exists. (so that we don't see any log with new  key.
>              2) revert to the the old versions of service.properties and
> 	         verify.key, this will also remove the
> 	         derby.storage.ReEncryptionLog property.
>              3) verify the user entered old key/password using the
>                verify.key/or the key in the service.properties. And then
>                proceed with recovery.
>              4) recovery will undo the partially completed re-encryption, 
>                 this will bring the containers back to versions of 
>                 the old encryption key.
>          }
>   } else 
>   {
>      if (reEncryptionLog == 0) 
>      {
>         // crash occurred during cleanup; re-encryption was complete.
>        call performcleanup();
>      }
>   }      
> If the above algorithm  works, Other minor issues:
> 1) instead of having service.properties.old file name the specific properties 
>    as new or old and set them accordingly. 
> 2) instead of storing the log file to delete incase of crash in the
>    as a property (derby.storage.ReEncryptionLog) in the
>    service.properties. Store it in the log.ctrl file; There is only one LONG
>    spare remaining , not sure if worth using for this case. 
> I would really appreciate if some one can take a look at the above algorithm 
> and verify if it really works or if there are any suggestions solve in
> different way, that will be great too. 
> Thanks
> -suresh
>>allow the encrypting of an existing unencrypted db and allow the re-encrypting of
an existing encrypted db
>>                Key: DERBY-1156
>>                URL: http://issues.apache.org/jira/browse/DERBY-1156
>>            Project: Derby
>>         Issue Type: Improvement
>>         Components: Store
>>   Affects Versions:
>>           Reporter: Mike Matrigali
>>        Assigned To: Suresh Thalamati
>>           Priority: Minor
>>            Fix For:
>>        Attachments: encryptspec.html, reencrypt_1.diff, reencrypt_2.diff, reencrypt_3.diff,
>>encrypted database to be re-encrypted with a new password.
>>Here are some ideas for an initial implementation.
>>The easiest way to do this is to make sure we have exclusive access to the
>>data and that no log is required in the new copy of the db.  I want to avoid
>>the log as it also is encrypted.  Here is my VERY high level plan:
>>1) Force exclusive access by putting all the work in the low level store,
>>   offline boot method.  We will do redo recovery as usual, but at the end
>>   there will be an entry point to do the copy/encrypt operation.
>>copy/encrypt process:
>>0) The request to encrypt/re-encrypt the db will be handled with a new set
>>   of url flags passed into store at boot time.  The new flags will provide
>>   the same inputs as the current encrypt flags.  So at high level the
>>   request will be "connect db old_encrypt_url_flags; new_encrypt_url_flags".
>>   TODO - provide exact new flag syntax.
>>1) Open a transaction do all logged work to do the encryption.  All logging
>>   will be done with existing encryption.
>>2) Copy and encrypt every db file in the database.  The target files will
>>   be in the data directory.  There will be a new suffix to track the new
>>   files, similar to the current process used for handling drop table in
>>   a transaction consistent manner without logging the entire table to the log.
>>   Entire encrypted destination file is guaranteed synced to disk before
>>   transaction commits.  I don't think this part needs to be logged.
>>   Files will be read from the cache using existing mechanism and written
>>   directly into new encrypted files (new encrypted data does not end up in
>>   the cache).
>>3) Switch encrypted files for old files.  Do this under a new log operation
>>   so the process can be correctly rolled back if the encrypt db operation
>>   transaction fails.  Rollback will do file at a time switches, no reading
>>   of encrypted data is necessary.
>>4) log a "change encryption of db" log record, but do not update
>>   system.properties with the change.
>>5) commit transaction.
>>6) update system.properties and sync changes.
>>7) TODO - need someway to handle crash between steps 5 and 6.
>>6) checkpoint all data, at this point guaranteed that there is no outstanding
>>   transaction, so after checkpoint is done there is no need for the log.
>>o there probably should be something that catches a request to encrypt to
>>  whatever db was already encrypted with.

View raw message