hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3381) INode interlinks can multiply effect of memory leaks
Date Tue, 13 May 2008 21:29:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596539#action_12596539
] 

Raghu Angadi commented on HADOOP-3381:
--------------------------------------

> The INode.parent reference is actually not necessary and waste space. It would be good
if we can remove it.

This is related but different issue. I am planning a patch that is suitable for 0.16 and up.
Even I thought parent link was removed earlier (to save memory).

Note that parent is not the only culprit. the links to children also have the similar negative
effects. 

We already iterate over all the files below a directory when it is deleted, so clearing a
few more references will be a straight fwd fix.


> INode interlinks can multiply effect of memory leaks
> ----------------------------------------------------
>
>                 Key: HADOOP-3381
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3381
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.16.0
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>
> Say a directory 'DIR' has a directory tree under it with 10000 files and directories.
Each INode keeps refs to parent and children. When DIR is deleted, memory wise we essentially
delete link from its parent (and delete the all the blocks from {{blocksMap}}). We don't modify
its children. This is ok since this will form an island of references and will be gc-ed. Thats
when everything is perfect. But if there is a bug that leaves a ref from a valid object (there
is a suspect, I will another jira) to even one of these 10000 files, it could hold up all
the INode and related objects. This can make a smaller mem leak many times more severe.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message