db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rick Hillegas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (DERBY-4127) ArrayIndexOutOfBoundsException in DatabaseMetaDataTest when run in Soft Upgrade mode
Date Fri, 10 Apr 2009 15:09:14 GMT

    [ https://issues.apache.org/jira/browse/DERBY-4127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12697833#action_12697833
] 

Rick Hillegas commented on DERBY-4127:
--------------------------------------

I have looked into this problem and would like to summarize what I have observed:

1) Simple test case

2) Scope of problem

3) Suspect code

4) Proposed solution (10.5)

5) Future improvements (10.6)


------- (1) Simple Test Case ---------------

Here is a simple way to reproduce this problem:

A) Hard-upgrade a 10.4 (or earlier) database to 10.5

B) Bring up the 10.5 network server

C) Run the following query through ij over the network:

    select * from sys.sysroutineperms

I didn't spend any time looking into the ugly network error. I wrote it off as just another
example of the obscure diagnostics which the network layer barks when it trips over a puzzling
condition. I believe that the underlying problem which muddled the network layer was the presence
of a null in a non-nullable column.

More specifically, the same query run through the embedded driver shows a corrupt permission
tuple. The PUBLIC permission granted to SYSCS_UPDATE_STATISTICS has a null GRANTOR column.
GRANTOR is a non-nullable column and it should contain the name of the database owner.

The tuple should be

( permissionID = someUUID, grantee = PUBLIC, grantor = databaseOwner, aliasID = updateStatisticsUUID,
grantOption = N )

but is instead

( permissionID = someUUID, grantee = PUBLIC, grantor = NULL, aliasID = updateStatisticsUUID,
grantOption = N )

An extra complication of this corruption occurs if a database is hard-upgraded from 10.0 to
10.5. In that case, the metadata holds two versions of the permission: the correct version
and the corrupt version.



------- (2) Scope of the Problem ---------------

I believe that this corrupt metadata is benign for the moment. That is, I don't believe that
this corruption causes Derby to malfunction in any serious way--other than exposing some brittleness
in the network layer. However, I think that corrupt metadata is a ticking timebomb and we
should not make the metadata more corrupt than it already is.

In investigating this problem, I discovered another example of this corruption which may have
crept into Derby databases already. The permission granted to SYSCS_INPLACE_COMPRESS_TABLE
has a null GRANTOR column if the database was hard-upgraded from 10.0 to 10.2 or later. Note
that the corruption does not appear in databases which underwent an intermediate upgrade to
10.1 before being hard-upgraded to a later release. So the corrupt upgrade trajectories are:

  10.0 -> 10.2
  10.0 -> 10.3
  10.0 -> 10.4

but the following trajectories are ok:

  10.0 -> 10.1 -> 10.2
  10.0 -> 10.1 -> 10.3
  10.0 -> 10.1 -> 10.4


------- (3) Suspect Code ---------------

To recap, we see two upgrade problems:

i) A NULL GRANTOR column in a permission tuple--seen in all hard-upgrades to 10.5.

ii) Redundant permission tuples--seen when you hard-upgrade from 10.0.

I believe these problems are caused by the following issues in Derby's metadata management:

i) We calculate the database owner AFTER upgrade. I think this was a deliberate decision.
Calculating the database owner AFTER upgrade gives you the correct owner when you hard-upgrade
to 10.2 or later. That is because 10.2 changes the database owner. However, this late calculation
means that the database owner is NULL during upgrade--and that is why we stuff NULL into the
GRANTOR column.

ii) We declare the list of PUBLIC routines twice: first in a tidy table and then again in
procedural logic. This redundant declaration results in multiple copies of the same permission.


------- (4) Proposed solution (10.5) ---------------

I recommend the following immediate changes to 10.5 and the trunk:

i) Calculate the database owner BEFORE the database is upgraded. Then recalculate the database
owner during upgrade if we have to run the 10.2 logic which changes the database owner.

ii) Declare the list of PUBLIC routines only once. I think this means removing one of the
existing declarations. Removing the tidy tables requires fewer code changes but results in
code that is harder to understand. Removing the procedural redundancies causes a lot of edits
but leaves you with a tidy declaration of the PUBLIC routines which you understand at a glance.
I vote for this solution.


------- (5) Future improvements (10.6) ---------------

Even after making the changes in (4), we are still left with the corrupt permission for SYSCS_INPLACE_COMPRESS_TABLE
in databases which were hard-upgraded as follows:

  10.0 -> 10.2
  10.0 -> 10.3
  10.0 -> 10.4

We could add some 10.5 upgrade logic which corrects this corruption. However, since I think
the corruption is benign for the moment and no-one has complained about it yet, I recommend
that we defer this correction until the 10.6 upgrade. I also recommend that we write a comprehensive
test which compares virgin metadata with hard-upgraded metadata. We should run this test on
all hard-upgrade trajectories. This may uncover other corruptions in Derby metadata. We should
correct those other corruptions in 10.6 too.


> ArrayIndexOutOfBoundsException in DatabaseMetaDataTest when run in Soft Upgrade mode
> ------------------------------------------------------------------------------------
>
>                 Key: DERBY-4127
>                 URL: https://issues.apache.org/jira/browse/DERBY-4127
>             Project: Derby
>          Issue Type: Bug
>          Components: Regression Test Failure
>    Affects Versions: 10.5.1.0
>         Environment: Windows Vista 64, Junit 3.8.2, Sun JDK 1.6.0_10
>            Reporter: Suran Jayathilaka
>            Assignee: Rick Hillegas
>            Priority: Blocker
>         Attachments: Test4127.java
>
>
> This bug was found when doing soft upgrade testing from Derby version 10.4.2.0 to 10.5.1.0
(RC1)
> Steps followed are as follows.
> 1. Run setEmbeddedCP.bat from version 10.4.2.0's bin folder
> 2. In a test folder run ij
> 3. create system/wombat database.
>     ij> connect 'jdbc:derby:system/wombat;create=true';
> 4. exit ij
> 5. Copy the 10.5.1.0 derby jars (from lib folder) and the derbyTesting.jar from 10.4.2.0
to the test folder and set classpath with them (including junit and ORO)
> 6. Run suites.All
>      java -Xmx512M -Xms512M -Dderby.tests.trace=true junit.textui.TestRunner org.apache.derbyTesting.functionTests.suites.All
> Result:
> Tests run: 10479, Failures: 56, Errors: 34 
> The stack trace is as follows.
> ---------------------------------------------------------------------------------------------------------
> 21) testGetColumnsReadOnly(org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest)java.lang.ArrayIndexOutOfBoundsException:
122
> 	at org.apache.derby.client.net.NetCursor.readFdocaBytes(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.parseSQLCAXGRP(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.parseSQLCAGRP(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.parseSQLCARD(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.calculateColumnOffsetsForRow_(Unknown Source)
> 	at org.apache.derby.client.am.Cursor.stepNext(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.scanDataBufferForEndOfData(Unknown Source)
> 	at org.apache.derby.client.net.NetResultSet.preClose_(Unknown Source)
> 	at org.apache.derby.client.am.ResultSet.closeX(Unknown Source)
> 	at org.apache.derby.client.am.ResultSet.close(Unknown Source)
> 	at org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.crossCheckGetColumnsAndResultSetMetaData(DatabaseMetaDataTest.java:1660)
> 	at org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.testGetColumnsReadOnly(DatabaseMetaDataTest.java:1357)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:102)
> 	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> 	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> 	at junit.extensions.TestSetup.run(TestSetup.java:25)
> 	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> 	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> 	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> 	at junit.extensions.TestSetup.run(TestSetup.java:25)
> 	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> 	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> 	at junit.extensions.TestSetup.run(TestSetup.java:25)
> 	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> 22) testGetColumnsModify(org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest)java.lang.ArrayIndexOutOfBoundsException:
122
> 	at org.apache.derby.client.net.NetCursor.readFdocaBytes(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.parseSQLCAXGRP(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.parseSQLCAGRP(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.parseSQLCARD(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.calculateColumnOffsetsForRow_(Unknown Source)
> 	at org.apache.derby.client.am.Cursor.stepNext(Unknown Source)
> 	at org.apache.derby.client.net.NetCursor.scanDataBufferForEndOfData(Unknown Source)
> 	at org.apache.derby.client.net.NetResultSet.preClose_(Unknown Source)
> 	at org.apache.derby.client.am.ResultSet.closeX(Unknown Source)
> 	at org.apache.derby.client.am.ResultSet.close(Unknown Source)
> 	at org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.crossCheckGetColumnsAndResultSetMetaData(DatabaseMetaDataTest.java:1660)
> 	at org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.testGetColumnsReadOnly(DatabaseMetaDataTest.java:1357)
> 	at org.apache.derbyTesting.functionTests.tests.jdbcapi.DatabaseMetaDataTest.testGetColumnsModify(DatabaseMetaDataTest.java:1374)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at org.apache.derbyTesting.junit.BaseTestCase.runBare(BaseTestCase.java:102)
> 	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> 	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> 	at junit.extensions.TestSetup.run(TestSetup.java:25)
> 	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> 	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> 	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> 	at junit.extensions.TestSetup.run(TestSetup.java:25)
> 	at junit.extensions.TestDecorator.basicRun(TestDecorator.java:24)
> 	at junit.extensions.TestSetup$1.protect(TestSetup.java:21)
> 	at junit.extensions.TestSetup.run(TestSetup.java:25)
> 	at org.apache.derbyTesting.junit.BaseTestSetup.run(BaseTestSetup.java:57)
> ----------------------------------------------------------------------------------
> I did not see anything in the derby.log from the tests that seemed to be related to this.
> Note that this test passed when run by itself, and also when run after a couple of failing
tests (namely org.apache.derbyTesting.functionTests.tests.jdbcapi.BlobClob4BlobTest and org.apache.derbyTesting.functionTests.tests.jdbcapi.UpdatableResultSetTest)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message