hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2899) hdfs:///mapredsystem directory not cleaned up after deallocation
Date Thu, 06 Mar 2008 10:52:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575650#action_12575650
] 

Hemanth Yamijala commented on HADOOP-2899:
------------------------------------------

We had a discussion with Sameer, and Devaraj on this issue. Logically, the mapred system directory
is something that the admin would set up in a static job tracker case. In that case, he would
be responsible for  managing this. In the HOD world, HOD is playing this role and generating
the mapred system directory. Hence, it feels like it should set up and clean it. However,
in the current design of HOD, this is not very easy to do in a short time. Therefore, I am
moving this to be fixed in Hadoop 0.17 in HOD. In the meantime, we can write a simple utility
to clean up these directories periodically. The impact of not doing it immediately is reduced
with such a utility / script.

> hdfs:///mapredsystem directory not cleaned up after deallocation 
> -----------------------------------------------------------------
>
>                 Key: HADOOP-2899
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2899
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/hod
>    Affects Versions: 0.16.0
>            Reporter: Luca Telloli
>            Assignee: Devaraj Das
>             Fix For: 0.17.0
>
>
> Each submitted job creates a hdfs:///mapredsystem directory, created by (I guess) the
hodring process. Problem is that it's not cleaned up at the end of the process; a use case
would be:
> - user A allocates a cluster, the hodring is svrX, so a /mapredsystem/srvX directory
is created
> - user A deallocates the cluster, but that directory is not cleaned up
> - user B allocates a cluster, and the first node chosen as hodring is svrX, so hodring
tries to write hdfs:///mapredsystem but it fails
> - allocation succeeds, but there's no hodring running; looking at
> 0-jobtracker/logdir/hadoop.log under the temporary directory I can read:
> 2008-02-26 17:28:42,567 WARN org.apache.hadoop.mapred.JobTracker: Error starting tracker:
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.fs.permission.AccessControlException:
Permission denied: user=B, access=WRITE, inode="mapredsystem":hadoop:supergroup:rwxr-xr-x
> I guess a possible solution would be to clean up those directories during the deallocation
process. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message