db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kristian Waagan (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (DERBY-3130) Reduce memory footprint of StoredRecordHeader
Date Mon, 22 Oct 2007 09:13:50 GMT

    [ https://issues.apache.org/jira/browse/DERBY-3130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536603
] 

kristwaa edited comment on DERBY-3130 at 10/22/07 2:12 AM:
------------------------------------------------------------------

I ran the patch with the load I experience problems with, and it fixes the problem for my
configuration.
As I mentioned earlier, the patch may not solve the problem for other configurations due to
the amount of heap available and the size of the Derby page cache.
In my case Derby was able to keep under 3.5 GB of heap with a page cache of size 262144 (which
with 4K pages would be around 1 GB + overhead).

Here's the start out the heap histogram:
num   #instances    #bytes  class name
--------------------------------------
  1:  33176246   1061639872  org.apache.derby.impl.store.raw.data.StoredRecordHeader
  2:  __683347   _961070736  [B
  3:  33402612   _801662688  org.apache.derby.impl.store.raw.data.RecordId
  4:  __226889   _136070680  [Lorg.apache.derby.impl.store.raw.data.StoredRecordHeader;
  5:  __226729    __43531968  org.apache.derby.impl.store.raw.data.StoredPage
  6:  __237426    __40619752  [C
  7:  __226653    __14505792  org.apache.derby.impl.store.access.btree.LeafControlRow
  8:  __226890     ___9075600  org.apache.derby.iapi.services.io.FormatIdInputStream
  9:  __226698     ___9068064  [Lorg.apache.derby.iapi.types.DataValueDescriptor;

The total is reported like this, which I interpret to mean that the heap usage is around 2.9
GB:
Total 70589681   -1158738416

Using the 48 byte version of StoredRecordHeader would add another 500 MB, causing the heap
to be filled to the rim (I haven't checked if these objects refer other objects as well) which
in turn effectively brings the application to an halt. With a 32-bit VM you can't give Derby
more heap, and the only option would be to reduce the page cache size.

+1 from me based on testing the patch  (I haven't reviewed the code changes).
In this specific case, 8 more bytes saved per instance would sum up to 253 MB.

      was (Author: kristwaa):
    I ran the patch with the load I experience problems with, and it fixes the problem for
my configuration.
As I mentioned earlier, the patch may not solve the problem for other configurations due to
the amount of heap available and the size of the Derby page cache.
In my case Derby was able to keep under 3.5 GB of heap with a page cache of size 262144 (which
with 4K pages would be around 1 GB + overhead).

Here's the start out the heap histogram:
num   #instances    #bytes  class name
--------------------------------------
  1:  33176246   1061639872  org.apache.derby.impl.store.raw.data.StoredRecordHe
ader
  2:    683347   961070736  [B
  3:  33402612   801662688  org.apache.derby.impl.store.raw.data.RecordId
  4:    226889   136070680  [Lorg.apache.derby.impl.store.raw.data.StoredRecordH
eader;
  5:    226729    43531968  org.apache.derby.impl.store.raw.data.StoredPage
  6:    237426    40619752  [C
  7:    226653    14505792  org.apache.derby.impl.store.access.btree.LeafControl
Row
  8:    226890     9075600  org.apache.derby.iapi.services.io.FormatIdInputStrea
m
  9:    226698     9068064  [Lorg.apache.derby.iapi.types.DataValueDescriptor;

The total is reported like this, which I interpret to mean that the heap usage is around 2.9
GB:
Total 70589681   -1158738416

Using the 48 byte version of StoredRecordHeader would add another 500 MB, causing the heap
to be filled to the rim (I haven't checked if these objects refer other objects as well) which
in turn effectively brings the application to an halt. With a 32-bit VM you can't give Derby
more heap, and the only option would be to reduce the page cache size.

+1 from me based on testing the patch  (I haven't reviewed the code changes).
In this specific case, 8 more bytes saved per instance would sum up to 253 MB.
  
> Reduce memory footprint of StoredRecordHeader
> ---------------------------------------------
>
>                 Key: DERBY-3130
>                 URL: https://issues.apache.org/jira/browse/DERBY-3130
>             Project: Derby
>          Issue Type: Improvement
>          Components: Store
>    Affects Versions: 10.4.0.0
>            Reporter: Knut Anders Hatlen
>            Priority: Minor
>         Attachments: SmallRecordsTest.java, srh.diff
>
>
> Derby's page cache often has a memory footprint that is much larger than pageSize*pageCacheSize.
One large contributor to the footprint is the array of StoredPageHeader objects in BasePage.
The memory consumed by these objects can be as large as, and sometimes even larger than, the
byte arrays containing the raw page data. (See for instance http://www.nabble.com/How-much-derby-need-memory--tf3307655.html.)
Reducing the size of the StoredPageHeader objects could therefore reduce Derby's memory footprint
significantly, especially if the page cache is large and contains many pages from tables with
small records or from indices.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message