directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Emmanuel Lecharny (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DIRSERVER-2053) Sometimes apacheds will throw OutOfMemoryException
Date Wed, 04 Mar 2015 15:55:05 GMT

    [ https://issues.apache.org/jira/browse/DIRSERVER-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347048#comment-14347048
] 

Emmanuel Lecharny commented on DIRSERVER-2053:
----------------------------------------------

I just found a cycle in the free page list, there may be more cycle elsewhere. I suspect that
is due to some concurrent writes, while loading the data in the server.

Let me explain how all this works.

The {{RecordManager}} is resonspible for writing pages on disk. It also manages {{B-trees}}
(in your case, there are 18 {{B-trees}}). There are a few meta-data that are used for that
:
- the {{RecordManager Header}}
- the {{B-tree of B-trees}}  aka {{BoB}}
- the {{Copied Pages B-tree}} aka {{CPB}} (not currently used)
- the free page lists

Each {{B-tree}} is using some meta-data too :
- the {{BtreeHeader}}
- the {{BtreeInfo}}

Let's get back to the {{RecordManager Header}}, which is the very first page of the file.
It contains  :

{code}
     * +---------------------+
     * | PageSize            | 4 bytes : The size of a physical page (default to 4096)
     * +---------------------+
     * | NbTree              | 4 bytes : The number of managed B-trees (at least 1)
     * +---------------------+
     * | FirstFree           | 8 bytes : The offset of the first free page
     * +---------------------+
     * | current BoB offset  | 8 bytes : The offset of the current B-tree of B-trees
     * +---------------------+
     * | previous BoB offset | 8 bytes : The offset of the previous B-tree of B-trees
     * +---------------------+
     * | current CP offset   | 8 bytes : The offset of the current CopiedPages B-tree
     * +---------------------+
     * | previous CP offset  | 8 bytes : The offset of the previous CopiedPages B-tree
     * +---------------------+
{code}

When we start the system, we will read this page. This is also the page that gets updated
(see {{RecordManager.updateRecordManagerHeader}}) every time we write something in any B-tree
(as it's a single page, it will always be written completely on disk,so it will always reflect
a consistent state. Still, we may have intermediate states, that is the reason why we have
current and previous offsets for the BOB and CPB).

The {{FreePageList}} links all the PageIOs that can be reused when we need to write some data.
They are all linked together, newly freed pages are added at the beginning of the list, the
new first free page offset is written in the {{RecordManager Header}}, when it's done (see
{{RecordManager.free()}}).

The {{BoB}} B-tree is the 'directory' of all the existing B-trees we manage. It contains all
the existing B-trees, with all the existing revisions. Ie, if we have a B-tree with 3 revisions,
then we will have 3 elements stored in the {{BoB}} B-tree.
This B-tree can grow quite quickly, if we don't remove older revisions that are not anymore
in use. Anyway, associated with a Tuple<Btree name, revision> is a {{BTreeHeader}}.

The {{BTreeHeader}} data structure is simple : it's a page containing the following informations
:
{code}
     * +------------+
     * | revision   | The B-tree revision
     * +------------+
     * | nbElems    | The B-tree number of elements
     * +------------+
     * | rootPage   | The root page offset
     * +------------+
     * | BtreeInfo  | The B-tree info offset
     * +------------+
{code}

The {{BtreeInfo}} is not really interesting, it contains static info about a given B-tree,
like its name, the key and value serializers FQCN and such things. In any case, it's written
once (when a B-tree is added), and read once (when we initialize a given B-tree), but never
ever modified.

The {{RootPage}} is the top level page of a B-tree. It can be a {{Node}} or a {{Leaf}}. If
it's a {{Node}}, then obviously it has children.

Let's go a bit deeper now. The file is split in pages which size is fixed (default to 512bytes).
That means we can only write only 512 bytes in it, so when we have to write more, then we
have to use more than one page. We have the possibility to link pages together to make it
possible to store more than 512bytes. The {{PageIO}} class is used to store the physical pages,
the first 8 bytes is a offset to the next page, or -1 ({{NO_PAGE}}) if this is the last page.
The very first page also contain the size of the data that has been put in all those linked
pages (from 0 to 2G bytes of possible data...).




> Sometimes apacheds will throw OutOfMemoryException
> --------------------------------------------------
>
>                 Key: DIRSERVER-2053
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-2053
>             Project: Directory ApacheDS
>          Issue Type: Bug
>          Components: ldap
>    Affects Versions: 2.0.0-M19
>         Environment: CentOS 6.5 with apacheds M19 installed.
>            Reporter: linzhao
>            Priority: Blocker
>
> After injecting some data into apacheds and restart it. Sometimes the apacheds can't
be started, it will throw OutOfMemoryException. The data entries will less than 600. Exception
list below:
> INFO   | jvm 1    | 2015/03/04 23:52:09 | Dumping heap to /opt/polycom/apacheds-dumps/java_pid24024.hprof
...
> INFO   | jvm 1    | 2015/03/04 23:52:19 | Heap dump file created [2094548383 bytes in
9.597 secs]
> INFO   | jvm 1    | 2015/03/04 23:52:19 | Error in WrapperListener.start callback.  java.lang.OutOfMemoryError:
GC overhead limit exceeded
> INFO   | jvm 1    | 2015/03/04 23:52:19 | java.lang.OutOfMemoryError: GC overhead limit
exceeded
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.mavibot.btree.RecordManager.fetchPage(RecordManager.java:3045)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.mavibot.btree.RecordManager.readPageIOs(RecordManager.java:797)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.mavibot.btree.RecordManager.deserialize(RecordManager.java:987)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.mavibot.btree.PersistedPageHolder.fetchElement(PersistedPageHolder.java:133)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.mavibot.btree.PersistedPageHolder.getValue(PersistedPageHolder.java:113)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.mavibot.btree.AbstractPage.get(AbstractPage.java:252)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.mavibot.btree.AbstractBTree.get(AbstractBTree.java:505)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.mavibot.btree.PersistedBTree.get(PersistedBTree.java:43)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.partition.impl.btree.mavibot.MavibotTable.get(MavibotTable.java:317)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.partition.impl.btree.mavibot.MavibotIndex.forwardLookup(MavibotIndex.java:305)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.partition.impl.btree.mavibot.MavibotIndex.forwardLookup(MavibotIndex.java:58)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.getEntryId(AbstractBTreePartition.java:2473)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.xdbm.search.impl.DefaultSearchEngine.computeResult(DefaultSearchEngine.java:123)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.search(AbstractBTreePartition.java:1141)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.shared.partition.DefaultPartitionNexus.search(DefaultPartitionNexus.java:624)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.shared.ReferralManagerImpl.init(ReferralManagerImpl.java:178)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.shared.ReferralManagerImpl.<init>(ReferralManagerImpl.java:86)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.referral.ReferralInterceptor.init(ReferralInterceptor.java:213)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.DefaultDirectoryService.initInterceptors(DefaultDirectoryService.java:685)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1818)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:1244)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.ApacheDsService.initDirectoryService(ApacheDsService.java:323)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.ApacheDsService.start(ApacheDsService.java:182)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(ApacheDsTanukiWrapper.java:72)
> INFO   | jvm 1    | 2015/03/04 23:52:19 |       at org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)
> STATUS | wrapper  | 2015/03/04 23:52:20 | <-- Wrapper Stopped



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message