hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4879) Add "blocked ArrayList" collection to avoid CMS full GCs
Date Thu, 06 Jun 2013 17:33:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13677274#comment-13677274

Todd Lipcon commented on HDFS-4879:

bq. Given that the list is short lived, why not just use linked list. I don't think the cost
of an object and next pointer should be a big deal for a short lived object.

In the case of a delete of 25M blocks, like we saw, the linked list is going to be significantly
bigger than the ArrayList. I think each Node object takes up 64 bytes, right? So the short
lived linked list would be 1.6GB instead of 400MB, which is likely to push it out of the young
generation. It also has worse locality of access, causing many more CPU cache misses to traverse

bq. Jonathan: Why not implement an actual j.u.List, e.g. via j.u.AbstractList?
bq. Daryn: I think this is a great change, but agree that ChunkedArrayList should ideally
be a full-fledged list. We may find this list implementation to be useful in other places,
which is a benefit over using an actual linked list.

I started out down this path, but given that the target use cases today only require accumulating
entries and then enumerating them, I didn't want to add a bunch of unused code for future
use cases we haven't found yet. I'd prefer to commit this simple fix for now, since it addresses
a real problem, and then if someone finds a further use case for it later which requires all
of the other List features, we can always add them. It's a private internal API, so I don't
see any problem expanding it later.

bq. Consider removing multiple calls to addChunk() to seed the main list by folding the logic
into add? It could add a new chunk if the list is either empty, or the existing full chunk

I'm not following what you mean here. Which code path are you talking about?

bq. Why does each additional chunk's capacity quadruple? If necessary, it would be more understandable
to multiple by 4.

It is actually tripling, which I thought was mirroring what ArrayList does. But in fact, it
looks like ArrayList grows by 1.5x each time:

    private void grow(int minCapacity) {
        // overflow-conscious code
        int oldCapacity = elementData.length;
        int newCapacity = oldCapacity + (oldCapacity >> 1);

I'll switch to that method here.

Will also make the other changes suggested above in the next revision.

> Add "blocked ArrayList" collection to avoid CMS full GCs
> --------------------------------------------------------
>                 Key: HDFS-4879
>                 URL: https://issues.apache.org/jira/browse/HDFS-4879
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 3.0.0, 2.0.4-alpha
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hdfs-4879.txt, hdfs-4879.txt
> We recently saw an issue where a large deletion was issued which caused 25M blocks to
be collected during {{deleteInternal}}. Currently, the list of collected blocks is an ArrayList,
meaning that we had to allocate a contiguous 25M-entry array (~400MB). After a NN has been
running for a long amount of time, the old generation may become fragmented such that it's
hard to find a 400MB contiguous chunk of heap.
> In general, we should try to design the NN such that the only large objects are long-lived
and created at startup time. We can improve this particular case (and perhaps some others)
by introducing a new List implementation which is made of a linked list of arrays, each of
which is size-limited (eg to 1MB).

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message