db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Pendleton (JIRA)" <j...@apache.org>
Subject [jira] Commented: (DERBY-3219) Group by query with many aggregate columns and case statements fails with: ERROR XSDA7: Restore of a serializable or SQLData object of class , attempted to read more data than was originally stored
Date Wed, 21 May 2008 02:42:55 GMT

    [ https://issues.apache.org/jira/browse/DERBY-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598541#action_12598541
] 

Bryan Pendleton commented on DERBY-3219:
----------------------------------------

Hi Knut, Thomas, I'm still confused as to why the repro test case that I
wrote isn't reliably demonstrating the problem in your environment.

Would it be possible for you to run the 'repro.java' test
program with  -Dderby.debug.true=SortTuning and have a look in your derby.log
to see if you are seeing the same sort of values I am seeing?

In the original repro (select * from pivotview), we encounter some
memory-overflow-sizing code in ExternalSortFactory.createSort:

    // if we have some idea on row size, set sort approx 1 meg of
    // memory sort.
    if (estimatedRowSize > 0)
    {
    	// 
    	// for each column, there is a reference from the key array and
    	//   the 4 bytes reference to the column object plus 12 bytes
    	//   tax on the  column object  
    	// for each row, SORT_ROW_OVERHEAD is the Node and 4 bytes to
    	// point to the column array and 4 for alignment
    	//
    	estimatedRowSize += SORT_ROW_OVERHEAD +
			(template.length*(4+12)) + 8; 
    	sortBufferMax = DEFAULT_MEM_USE/estimatedRowSize;
    }
    else
    {
    	sortBufferMax = defaultSortBufferMax;
    }
			
    // if there are barely more rows than sortBufferMax, use 2
    // smaller runs of similar size instead of one larger run
    //
    // 10% slush is added to estimated Rows to catch the case where
    // estimated rows underestimate the actual number of rows by 10%.
    //
    if (estimatedRows > sortBufferMax &&
    	(estimatedRows*1.1) < sortBufferMax*2)
    	sortBufferMax = (int)(estimatedRows/2 + estimatedRows/10);

In the original reproduction case, the variables have the following values:
 - estimatedRowSize: 0
 - estimatedRows: 1407
 - defaultSortBufferMax: 1024

With these values, the last "if" statement above evalutes to TRUE, and
this causes us to compute a SortBuffer size of 843 rows, and since the
actual result has 2065 rows, we overflow the in-memory sort buffer and
we externalize the MaxMinAggregator instances, leading to the problem.

In the 'repro.java' case that I attached to DERBY-3219, when I get to
ExternalSortFactory.createSort, the variables have the values:
 - estimatedRowSize: 0
 - estimatedRows: 30
 - defaultSortBufferMax: 1024

With these values, the "if" statement is NOT true, and so we leave
sortBufferMax set to 1024, and since the actual result has 2000 rows,
we again overflow the in-memory sort buffer and trigger the problem.

In a SANE build, I can observe this processing by running with
-Dderby.debug.true=SortTuning, and then looking in derby.log for
lines like:

DEBUG SortTuning OUTPUT: sortBufferMax = 1024 estimatedRows = 30 estimatedRowSize = 0 defaultSortBufferMax
= 1024
DEBUG SortTuning OUTPUT: Growing sortBuffer dynamically, current sortBuffer capacity= 1023
estimatedMemoryUsed = 8587736 currentTotalMemory = 23293952 currentFreeMemory = 10340912 numcolumn
= 4 real per row memory = 8394



> Group by query with many aggregate columns and case statements fails with: ERROR XSDA7:
Restore of a serializable or SQLData object of class , attempted to read more data than was
originally stored
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: DERBY-3219
>                 URL: https://issues.apache.org/jira/browse/DERBY-3219
>             Project: Derby
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 10.3.1.4
>            Reporter: Stan Bradbury
>            Assignee: Bryan Pendleton
>         Attachments: maxminPatch.diff, patchWithTest.diff, pivotView.zip, repro.java,
testWithMemControls.diff
>
>
> using the attached database (v10.3) - " select * from pivotview " fails with the stack
trace below.  A view (pivotview_ok) created on a subset of the columns in pivotview executes
fine.  Adding one column back into pivotview_ok causes failures most of the time.  See attached
for view definitions.
> 2007-11-21 00:58:49.421 GMT Thread[main,5,main] (XID = 2734422), (SESSIONID = 0), (DATABASE
= pivotview), (DRDAID = null), Failed Statement is: select * from pivotview
> ERROR XSDA7: Restore of a serializable or SQLData object of class , attempted to read
more data than was originally stored
> 	at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
> 	at org.apache.derby.impl.store.raw.data.StreamFileContainer.fetchNext(Unknown Source)
> 	at org.apache.derby.impl.store.raw.data.StreamFileContainerHandle.fetchNext(Unknown
Source)
> 	at org.apache.derby.impl.store.access.sort.MergeScan.mergeARow(Unknown Source)
> 	at org.apache.derby.impl.store.access.sort.MergeScan.init(Unknown Source)
> 	at org.apache.derby.impl.store.access.sort.MergeSort.openSortScan(Unknown Source)
> 	at org.apache.derby.impl.store.access.RAMTransaction.openSortScan(Unknown Source)
> 	at org.apache.derby.impl.sql.execute.GroupedAggregateResultSet.loadSorter(Unknown Source)
> 	at org.apache.derby.impl.sql.execute.GroupedAggregateResultSet.openCore(Unknown Source)
> 	at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.openCore(Unknown Source)
> 	at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.open(Unknown Source)
> 	at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.ij.executeImmediate(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.utilMain.doCatch(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.utilMain.runScriptGuts(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.utilMain.go(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.Main.go(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.Main.mainCore(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.Main14.main(Unknown Source)
> 	at org.apache.derby.tools.ij.main(Unknown Source)
> Caused by: java.io.EOFException
> 	at java.io.DataInputStream.readBoolean(DataInputStream.java:248)
> 	at org.apache.derby.impl.sql.execute.MaxMinAggregator.readExternal(Unknown Source)
> 	at org.apache.derby.iapi.services.io.FormatIdInputStream.readObject(Unknown Source)
> 	at org.apache.derby.iapi.types.UserType.readExternal(Unknown Source)
> 	... 22 more
> ============= begin nested exception, level (1) ===========
> java.io.EOFException
> 	at java.io.DataInputStream.readBoolean(DataInputStream.java:248)
> 	at org.apache.derby.impl.sql.execute.MaxMinAggregator.readExternal(Unknown Source)
> 	at org.apache.derby.iapi.services.io.FormatIdInputStream.readObject(Unknown Source)
> 	at org.apache.derby.iapi.types.UserType.readExternal(Unknown Source)
> 	at org.apache.derby.impl.store.raw.data.StreamFileContainer.fetchNext(Unknown Source)
> 	at org.apache.derby.impl.store.raw.data.StreamFileContainerHandle.fetchNext(Unknown
Source)
> 	at org.apache.derby.impl.store.access.sort.MergeScan.mergeARow(Unknown Source)
> 	at org.apache.derby.impl.store.access.sort.MergeScan.init(Unknown Source)
> 	at org.apache.derby.impl.store.access.sort.MergeSort.openSortScan(Unknown Source)
> 	at org.apache.derby.impl.store.access.RAMTransaction.openSortScan(Unknown Source)
> 	at org.apache.derby.impl.sql.execute.GroupedAggregateResultSet.loadSorter(Unknown Source)
> 	at org.apache.derby.impl.sql.execute.GroupedAggregateResultSet.openCore(Unknown Source)
> 	at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.openCore(Unknown Source)
> 	at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.open(Unknown Source)
> 	at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> 	at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.ij.executeImmediate(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.utilMain.doCatch(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.utilMain.runScriptGuts(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.utilMain.go(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.Main.go(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.Main.mainCore(Unknown Source)
> 	at org.apache.derby.impl.tools.ij.Main14.main(Unknown Source)
> 	at org.apache.derby.tools.ij.main(Unknown Source)
> ============= end nested exception, level (1) ===========
> Cleanup action completed

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message