hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dick King (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1914) TrackerDistributedCacheManager never cleans its input directories
Date Sat, 31 Jul 2010 00:13:18 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894213#action_12894213
] 

Dick King commented on MAPREDUCE-1914:
--------------------------------------

The regression passes, except for the usual {{TestTaskTrackerLocalization}} .

> TrackerDistributedCacheManager never cleans its input directories
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-1914
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1914
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Dick King
>            Assignee: Dick King
>         Attachments: MAPREDUCE-1914--2010-07-30--1336.patch
>
>
> When we localize a file into a node's cache, it's installed in a directory whose subroot
is a random {{long}} .  These {{long}} s all sit in a single flat directory [per disk, per
cluster node].  When the cached file is no longer needed, its reference count becomes zero
in a tracking data structure.  The file then becomes eligible for deletion when the total
amount of space occupied by cached files exceeds 10G [by default] or the total number of such
files exceeds 10K.
> However, when we delete a cached file, we don't delete the directory that contains it;
this importantly includes the elements of the flat directory, which then accumulate until
they reach a system limit, 32K in some cases, and then the node stops working.
> We need to delete the flat directory when we delete the localized cache file it contains.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message