directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Karasulu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (DIRSERVER-1459) Adding members to a groupOfNames results in polynomial increase in JDBM partition size
Date Sun, 24 Jan 2010 12:30:22 GMT

    [ https://issues.apache.org/jira/browse/DIRSERVER-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804233#action_12804233
] 

Alex Karasulu commented on DIRSERVER-1459:
------------------------------------------

I've been working on this issue for some time and have found out exactly why it is resulting.
 First this is a general problem that occurs irregardless of uniqueMember attributes.  It's
occurring due to the inability of JDBM to reclaim blocks used by secondary BTree objects for
storing large numbers of values for duplicate keys.  The JdbmTable wrapper is designed to
switch at some configurable threshold to begin using embedded BTrees to store values instead
of serializing a TreeSet/ArraySet container data structure.

When this happens and a nested BTree is used, the space used cannot be reclaimed by JDBM when
the BTree is deleted.  Plus modify operations for some reason are deleting and adding the
values causing the data to be rewritten.  This might be due to yet another issue all together
in the design of the modify operation.  This will need to be investigated.  However the need
to free blocks used by embedded secondary BTrees used for duplicate key value storage is a
must.

I am looking at JDBM right now to see how we can make sure this does in fact happen properly.
 This is not a small matter and is critical for proper operation.  I will keep you informed
as this progresses.

> Adding members to a groupOfNames results in polynomial increase in JDBM partition size
> --------------------------------------------------------------------------------------
>
>                 Key: DIRSERVER-1459
>                 URL: https://issues.apache.org/jira/browse/DIRSERVER-1459
>             Project: Directory ApacheDS
>          Issue Type: Bug
>    Affects Versions: 1.5.5
>         Environment: Any (tested on Linux and Mac OS X)
>            Reporter: Ben Hoyt
>            Assignee: Alex Karasulu
>            Priority: Blocker
>             Fix For: 1.5.6
>
>         Attachments: DIRSERVER-1459.tar.gz, screenshot-1.jpg
>
>
> I noticed a polynomial increase JDBM partition size and therefore disk usage when adding
users to groups in my ApacheDS instance.  The vast majority of the usage (95+% once you hit
a couple thousand users) is in workingDirectory/partitionId/master.db  
> Further testing showed that simply adding a user is linear, as one would expect, and
as 'apacheds-tools capacity' confirms.  It is only when a user is made a member of a group
that the JDBM partition size shoots up.
> Example statistics:
> Add 16,000 users - JDBM partition size = ~70 megabytes
> Now add those same 16,000 users to a single group (all in the same group) - JDBM partition
size = ~19 GIGABYTES
> I'll work to attach a test case and some more numbers from my tests

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message