Mailing-List: contact derby-user-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Derby Discussion" <derby-user@db.apache.org>
Received-SPF: pass (athena.apache.org: local policy)
From: Pavel Bortnovskiy <pbortnovskiy@jefferies.com>
To: Derby Discussion <derby-user@db.apache.org>
Subject: RE: NPE
Thread-Topic: NPE
Thread-Index: Ac3W+ecsZdqZiKRhQWugEmX3FEvv8gAMNsCAAAnwmdD//9TugP/+/sNA
Date: Tue, 11 Dec 2012 16:53:05 +0000
Message-ID: 
 <619F13B2042F204E8E8E93D73870255809DF6C1D@EXJSQUSDAG04.ad.jefco.com>
References: 
 <619F13B2042F204E8E8E93D73870255809DF5756@EXJSQUSDAG04.ad.jefco.com>
 <50C62443.1000306@oracle.com>
 <619F13B2042F204E8E8E93D73870255809DF579D@EXJSQUSDAG04.ad.jefco.com>
 <50C642D7.6010405@oracle.com>
In-Reply-To: <50C642D7.6010405@oracle.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Dag, et al. if I may add a bit of color to what's happening...
I've done lots of various tests (unfortunately, still unable to distill it =
down to an easy-to-reproduce tester).
And what I discovered is baffling...

The issue seems to occur (with 100% certainty) under the following circumst=
ances:
(1) two db connections - one to Derby (in-memory only mode) and Oracle
(2) a prepared statement is executed (such as "select * from <table>") agai=
nst Derby
(3) Oracle is queried for tables with:
final ResultSet resultSet =3D databaseMetaData.getTables(null, null, null, =
new String[]{"TABLE"});
(4) if the result set from (2) is iterated over AFTER the (3) then Derby fa=
ils with the described NPE; however, if it is iterated over BEFORE (3), the=
n OK!

It may also be worth noting that my resultsets are made "rewindable", i.e. =
with:
                statement =3D connection.prepareStatement(
                        sql,
                        ResultSet.TYPE_SCROLL_INSENSITIVE,
                        ResultSet.CONCUR_READ_ONLY);


Finally: although I am completely perplexed as to what's causing this probl=
em, I found a work-around which, in my estimation, will have minimal impact=
: at a particular point in my code (but effectively before calling getTable=
s() in (3)), the result set from (2) is iterated to the first record and th=
en rewound:

(not exact code, but that's what effectively is occurring):

If (resultSet.next()) {
   resultSet.beforeFirst();
}

As odd as it sounds, it seems to have fixed the issue...

I am planning to return to this as time allows. Then I may be able to creat=
e a stand-alone tester.
But if you have any thoughts as to what might be going on, I would be most =
appreciative.

Regards,
Pavel.

-----Original Message-----
From: Dag Wanvik [mailto:dag.wanvik@oracle.com]
Sent: Monday, December 10, 2012 3:15 PM
To: derby-user@db.apache.org
Subject: Re: NPE


On 10.12.2012 19:21, Pavel Bortnovskiy wrote:
> I upgraded to 10.9.1.0 and rerun with debug libraries (please disregard m=
y previous message).
>
> Mon Dec 10 13:19:44 EST 2012
> Thread[DatabasePublisher[BOND_WRITER],5,main] (XID =3D 289), (SESSIONID =
=3D 5), (DATABASE =3D memory:test), (DRDAID =3D null), Failed Statement is:=
 select *  from READER java.lang.NullPointerException
>         at org.apache.derby.impl.store.access.conglomerate.GenericScanCon=
troller.reopenAfterEndTransaction(GenericScanController.java:1048)
>         at

It looks like "container" is null, cf. the line 1048:

        case SCAN_HOLD_INIT:
***>         reusableRecordIdSequenceNumber =3D
                container.getReusableRecordIdSequenceNumber();

i.e. has been closed, cf the assignment just above:

        ContainerHandle container =3D open_conglom.reopen();

The call to reopen just returns null if OpenConglomerate has been closed so=
mehow.

Smells like a race condition of sorts, maybe some missing synchronization i=
n the Derby code.
If you are able to post a self contained repro program, although this might=
 be tricky if this is indeed a race, this would increase your chances of ha=
ving this fixed soon..

Thanks,
Dag


 In any case the NullPointerException shoudln't be seen by the application =
code, so this is a bug.


> org.apache.derby.impl.store.access.conglomerate.GenericScanController.fet=
chRows(GenericScanController.java:622)
>         at org.apache.derby.impl.store.access.heap.HeapScan.fetchNextGrou=
p(HeapScan.java:322)
>         at org.apache.derby.impl.sql.execute.BulkTableScanResultSet.reloa=
dArray(BulkTableScanResultSet.java:353)
>         at org.apache.derby.impl.sql.execute.BulkTableScanResultSet.getNe=
xtRowCore(BulkTableScanResultSet.java:308)
>         at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.get=
NextRowCore(ProjectRestrictResultSet.java:261)
>         at org.apache.derby.impl.sql.execute.ScrollInsensitiveResultSet.g=
etNextRowFromSource(ScrollInsensitiveResultSet.java:801)
>         at org.apache.derby.impl.sql.execute.ScrollInsensitiveResultSet.g=
etNextRowCore(ScrollInsensitiveResultSet.java:518)
>         at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getN=
extRow(BasicNoPutResultSetImpl.java:478)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(EmbedRe=
sultSet.java:432)
>         at org.apache.derby.impl.jdbc.EmbedResultSet.next(EmbedResultSet.=
java:376)
>         at QueryResults$1.hasNext(QueryResults.java:268)
>         at DatabasePublisher.process(DatabasePublisher.java:318)
>         at java.lang.Thread.run(Thread.java:662)
>
> -----Original Message-----
> From: Rick Hillegas [mailto:rick.hillegas@oracle.com]
> Sent: Monday, December 10, 2012 1:05 PM
> To: derby-user@db.apache.org
> Subject: Re: NPE
>
> Hi Pavel,
>
> Can you reproduce this problem with the debug version of Derby? That woul=
d give us line numbers in the stack trace.
>
> Thanks,
> -Rick
>
> On 12/10/12 9:23 AM, Pavel Bortnovskiy wrote:
>> Hello:
>>
>> While testing my application, which uses Derby in memory-only mode, I
>> get NPE inside Derby under "mysterious" circumstances.
>>
>> At this point, I can't even figure out what's triggering it and where
>> this could happen, but let me preempt my troubleshooting with some
>> stack traces. Perhaps someone, with good knowledge of Derby's code,
>> may offer some explanation. Here are some stack traces:
>>
>> Mon Dec 10 12:12:36 EST 2012 Thread[DatabasePublisher[WRITER],5,main]
>> (XID =3D 235), (SESSIONID =3D 5), (DATABASE =3D memory:test), (DRDAID =
=3D
>> null), Failed Statement is: select * from READER
>>
>> java.lang.NullPointerException
>>
>> at
>> org.apache.derby.impl.store.access.conglomerate.GenericScanController.
>> reopenAfterEndTransaction(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.store.access.conglomerate.GenericScanController.
>> fetchRows(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.store.access.heap.HeapScan.fetchNextGroup(Unkno
>> w
>> n Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.BulkTableScanResultSet.reloadArray(
>> U
>> nknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.BulkTableScanResultSet.getNextRowCo
>> r
>> e(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRow
>> C
>> ore(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.ScrollInsensitiveResultSet.getNextR
>> o
>> wFromSource(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.ScrollInsensitiveResultSet.getNextR
>> o
>> wCore(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(
>> U
>> nknown
>> Source)
>>
>> at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown
>> Source)
>>
>> at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
>>
>> at QueryResults$1.hasNext(QueryResults.java:268)
>>
>> at DatabasePublisher.process(DatabasePublisher.java:318)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>> or
>>
>> 2012-12-10 12:12:36,537 [DatabasePublisher[WRITER]] ERROR
>> QueryResults
>> - Unable to execute ResultSet.next() for Statement ""select * from
>> READER" [DERBY]"
>>
>> java.sql.SQLException: Java exception: ': java.lang.NullPointerException=
'.
>>
>> at
>> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unkn
>> o
>> wn
>> Source)
>>
>> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown
>> Source)
>>
>> at org.apache.derby.impl.jdbc.Util.javaException(Unknown Source)
>>
>> at
>> org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException
>> (
>> Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Un
>> k
>> nown
>> Source)
>>
>> at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
>> Source)
>>
>> at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unk
>> n
>> own
>> Source)
>>
>> at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown
>> Source)
>>
>> at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source)
>>
>> at QueryResults$1.hasNext(QueryResults.java:268)
>>
>> at DatabasePublisher.process(DatabasePublisher.java:318)
>>
>> at java.lang.Thread.run(Thread.java:662)
>>
>> Caused by: java.sql.SQLException: Java exception: ':
>> java.lang.NullPointerException'.
>>
>> at
>> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknow
>> n
>> Source)
>>
>> at
>> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransport
>> A
>> crossDRDA(Unknown
>> Source)
>>
>> ... 14 more
>>
>> Caused by: java.lang.NullPointerException
>>
>> at
>> org.apache.derby.impl.store.access.conglomerate.GenericScanController.
>> reopenAfterEndTransaction(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.store.access.conglomerate.GenericScanController.
>> fetchRows(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.store.access.heap.HeapScan.fetchNextGroup(Unkno
>> w
>> n Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.BulkTableScanResultSet.reloadArray(
>> U
>> nknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.BulkTableScanResultSet.getNextRowCo
>> r
>> e(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRow
>> C
>> ore(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.ScrollInsensitiveResultSet.getNextR
>> o
>> wFromSource(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.ScrollInsensitiveResultSet.getNextR
>> o
>> wCore(Unknown
>> Source)
>>
>> at
>> org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(
>> U
>> nknown
>> Source)
>>
>> ... 6 more
>>
>> One more thing I can share is that this seems to also be a timing
>> problem - if the code is executed slowly, line-by-line in the
>> debugger, then this problem doesn't occur. But if run at full speed,
>> then these exceptions are thrown. The problem is not intermittent and
>> easily reproduced 100% of the time on my end.
>>
>> Any thoughts would be greatly appreciated.
>>
>> Pavel.
>>
>>
>>
>>             Jefferies archives and monitors outgoing and incoming
>>             e-mail. The contents of this email, including any
>>             attachments, are confidential to the ordinary user of the
>>             email address to which it was addressed. If you are not
>>             the addressee of this email you may not copy, forward,
>>             disclose or otherwise use it or any part of it in any form
>>             whatsoever. This email may be produced at the request of
>>             regulators or in connection with civil litigation.
>>             Jefferies accepts no liability for any errors or omissions
>>             arising as a result of transmission. Use by other than
>>             intended recipients is prohibited. In the United Kingdom,
>>             Jefferies operates as Jefferies International Limited;
>>             registered in England: no. 1978621; registered office:
>>             Vintners Place, 68 Upper Thames Street, London EC4V 3BJ.
>>             Jefferies International Limited is authorised and
>>             regulated by the Financial Services Authority.
>>
>
> Jefferies archives and monitors outgoing and incoming e-mail. The content=
s of this email, including any attachments, are confidential to the ordinar=
y user of the email address to which it was addressed. If you are not the a=
ddressee of this email you may not copy, forward, disclose or otherwise use=
 it or any part of it in any form whatsoever. This email may be produced at=
 the request of regulators or in connection with civil litigation. Jefferie=
s accepts no liability for any errors or omissions arising as a result of t=
ransmission. Use by other than intended recipients is prohibited. In the Un=
ited Kingdom, Jefferies operates as Jefferies International Limited; regist=
ered in England: no. 1978621; registered office: Vintners Place, 68 Upper T=
hames Street, London EC4V 3BJ. Jefferies International Limited is authorise=
d and regulated by the Financial Services Authority.


Jefferies archives and monitors outgoing and incoming e-mail. The contents =
of this email, including any attachments, are confidential to the ordinary =
user of the email address to which it was addressed. If you are not the add=
ressee of this email you may not copy, forward, disclose or otherwise use i=
t or any part of it in any form whatsoever. This email may be produced at t=
he request of regulators or in connection with civil litigation. Jefferies =
accepts no liability for any errors or omissions arising as a result of tra=
nsmission. Use by other than intended recipients is prohibited. In the Unit=
ed Kingdom, Jefferies operates as Jefferies International Limited; register=
ed in England: no. 1978621; registered office: Vintners Place, 68 Upper Tha=
mes Street, London EC4V 3BJ. Jefferies International Limited is authorised =
and regulated by the Financial Services Authority.