Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Date: Fri, 5 Apr 2013 14:35:15 +0000 (UTC)
From: "Rick Hillegas (JIRA)" <jira@apache.org>
To: derby-dev@db.apache.org
Message-ID: <JIRA.12640149.1364843082405.116678.1365172515783@arcas>
In-Reply-To: <JIRA.12640149.1364843082405@arcas>
References: <JIRA.12640149.1364843082405@arcas>
Subject: [jira] [Updated] (DERBY-6136) Create a custom/optional tool for
 dumping the data in a corrupted database.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/DERBY-6136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rick Hillegas updated DERBY-6136:
---------------------------------

    Attachment: DataFileVTI.java

Attaching a new version of DataFileVTI. This version makes the following changes:

1) Skips records which are marked as deleted.

2) Turns errors into SQLWarnings so that problems will still be reported as reading continues. The warnings percolate up to the user only if you are using a codeline improved by derby-6151-01-aa-passBackWarnings.diff.

3) Removed dead code and reorganized to improve readability.

I have run DataFileVTI successfully against corrupt heaps. That is, against heaps which spew hex dumps when you try to select from them directly. Here is an ij example of using DataFileVTI to read a corrupt heap:

call syscs_util.syscs_register_tool
(
    'customTool',
    true,
    'RawDBReader', 
    'CONTROL11',
    'RAW11_',
    '/Users/rh161140/derby/mainline/corruptEncryptedDB',
    'bootPassword=mysecretpassword',
    'APP',
    null
);
0 rows inserted/updated/deleted
ij(CONNECTION2)> select * from raw11_app.t1 where a > 37 and a < 50;
A          |B                                                                                                                               |C                                                                                                                               
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
38         |abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrs&|abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrs&
39         |abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrs&|abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrs&
48         |abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrs&|abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrstuvwxyz0123456789abcdefghijklmnopqrs&
WARNING (no SQLState): Error reading field data. Offset = 366, length = 290, datatype = VARCHAR(1000): Field 2 in record 0 on page 5 in file c490.dat: java.io.UTFDataFormatException: null
WARNING (no SQLState): Error reading field data. Offset = 671, length = 1823514738, datatype = VARCHAR(1000): Field 1 in record 1 on page 5 in file c490.dat: java.io.IOException: Negative seek offset
WARNING (no SQLState): Error reading field data. Offset = 685, length = 36, datatype = VARCHAR(1000): Field 2 in record 1 on page 5 in file c490.dat: java.io.EOFException: null
WARNING (no SQLState): java.lang.ArrayIndexOutOfBoundsException: null
49         |abcdefghijklmnopqrstuvwxyz0123456789                                                                                            |abcdefghijklmnopqrstuvwxyz0123456789                                                                                            

4 rows selected


I think that these tools are ready for experimental use by others. Please tell me about the problems you find, so that I can improve these tools. Thanks.

                
> Create a custom/optional tool for dumping the data in a corrupted database.
> ---------------------------------------------------------------------------
>
>                 Key: DERBY-6136
>                 URL: https://issues.apache.org/jira/browse/DERBY-6136
>             Project: Derby
>          Issue Type: Improvement
>          Components: Tools
>    Affects Versions: 10.11.0.0
>            Reporter: Rick Hillegas
>         Attachments: DataFileVTI.java, DataFileVTI.java, DataFileVTI.java, DataFileVTI.java, dataFileVTI.sql, RawDBReader.java, RawDBReader.java
>
>
> It would be useful to have a tool for dumping the data in a corrupted database. This could start out as a custom tool. After we debug the tool and get some experience with it, we can consider promoting it to be a (possibly undocumented) optional tool which we ship with the product. I think the tool should have the following behavior:
> 1) The tool should not subvert the security of the corrupted database. If the corrupted database is password-protected, then you would need to present its DBO's credentials in order to use the tool. Naturally, an encryption key would have to be presented in order to decode an encrypted database.
> 2) The tool should not stop reading a table when it hits a corrupt record. Instead, the tool should soldier on and collect a list of warnings on bad records.
> Such a tool would be useful in situations where some part of a heap table is corrupt but the following heap conglomerates are intact:
> i) SYSSCHEMAS
> ii) SYSTABLES
> iii) SYSCONGLOMERATES
> iv) SYSCOLUMNS
> v) property conglomerate
> Such a tool would be useful for some situations where data can't be dumped even after you delete the log files in order to short-circuit recovery.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira