hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2899) [HOD] hdfs:///mapredsystem directory not cleaned up after deallocation
Date Wed, 19 Mar 2008 17:24:26 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Hemanth Yamijala updated HADOOP-2899:

    Attachment: 2899.1.patch

Attached patch adds code to the hodcleanup process to delete the mapred system directory.

- The hodring process passes arguments required for the deletion (process id of the job tracker,
namenode url, path to the hadoop command, and the name of the mapred system directory) to
the hodcleanup process. 
- The hodcleaup process first waits until the jobtracker is down. It waits by checking for
the job tracker process to end.
- Then it issues a hadoop dfs -fs namenode-url -rmr mapredsystemdir

Another change made in this patch is the way the namenode url is specified while copying logs
to the dfs as part of the hod cleanup process.

Added a few tests. However, they are not really complete. We need to refactor hod code a bit
to enable more unit testing - like providing the ability to create a mock simplecommand etc.
These can be handled later.

> [HOD] hdfs:///mapredsystem directory not cleaned up after deallocation 
> -----------------------------------------------------------------------
>                 Key: HADOOP-2899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2899
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/hod
>    Affects Versions: 0.16.0
>            Reporter: Luca Telloli
>            Assignee: Hemanth Yamijala
>             Fix For: 0.17.0
>         Attachments: 2899.1.patch
> Each submitted job creates a hdfs:///mapredsystem directory, created by (I guess) the
hodring process. Problem is that it's not cleaned up at the end of the process; a use case
would be:
> - user A allocates a cluster, the hodring is svrX, so a /mapredsystem/srvX directory
is created
> - user A deallocates the cluster, but that directory is not cleaned up
> - user B allocates a cluster, and the first node chosen as hodring is svrX, so hodring
tries to write hdfs:///mapredsystem but it fails
> - allocation succeeds, but there's no hodring running; looking at
> 0-jobtracker/logdir/hadoop.log under the temporary directory I can read:
> 2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker:
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.fs.permission.AccessControlException:
Permission denied: user=B, access=WRITE, inode="mapredsystem":hadoop:supergroup:rwxr-xr-x
> I guess a possible solution would be to clean up those directories during the deallocation

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message