[ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536603
]
Kristian Waagan commented on DERBY-3130:
----------------------------------------
I ran the patch with the load I experience problems with, and it fixes the problem for my
configuration.
As I mentioned earlier, the patch may not solve the problem for other configurations due to
the amount of heap available and the size of the Derby page cache.
In my case Derby was able to keep under 3.5 GB of heap with a page cache of size 262144 (which
with 4K pages would be around 1 GB + overhead).
Here's the start out the heap histogram:
num #instances #bytes class name
--------------------------------------
1: 33176246 1061639872 org.apache.derby.impl.store.raw.data.StoredRecordHe
ader
2: 683347 961070736 [B
3: 33402612 801662688 org.apache.derby.impl.store.raw.data.RecordId
4: 226889 136070680 [Lorg.apache.derby.impl.store.raw.data.StoredRecordH
eader;
5: 226729 43531968 org.apache.derby.impl.store.raw.data.StoredPage
6: 237426 40619752 [C
7: 226653 14505792 org.apache.derby.impl.store.access.btree.LeafControl
Row
8: 226890 9075600 org.apache.derby.iapi.services.io.FormatIdInputStrea
m
9: 226698 9068064 [Lorg.apache.derby.iapi.types.DataValueDescriptor;
The total is reported like this, which I interpret to mean that the heap usage is around 2.9
GB:
Total 70589681 -1158738416
Using the 48 byte version of StoredRecordHeader would add another 500 MB, causing the heap
to be filled to the rim (I haven't checked if these objects refer other objects as well) which
in turn effectively brings the application to an halt. With a 32-bit VM you can't give Derby
more heap, and the only option would be to reduce the page cache size.
+1 from me based on testing the patch (I haven't reviewed the code changes).
In this specific case, 8 more bytes saved per instance would sum up to 253 MB.
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
> Key: DERBY-3130
> URL: https://issues.apache.org/jira/browse/DERBY-3130
> Project: Derby
> Issue Type: Improvement
> Components: Store
> Affects Versions: 10.4.0.0
> Reporter: Knut Anders Hatlen
> Priority: Minor
> Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize.
One large contributor to the footprint is the array of StoredPageHeader objects in BasePage.
The memory consumed by these objects can be as large as, and sometimes even larger than, the
byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.)
Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint
significantly, especially if the page cache is large and contains many pages from tables with
small records or from indices.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
|