Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 3 Aug 2016 03:41:20 +0000 (UTC)
From: "mujunchao (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: <JIRA.12993427.1469758436000.207196.1470195680669@Atlassian.JIRA>
In-Reply-To: <JIRA.12993427.1469758436000@Atlassian.JIRA>
References: <JIRA.12993427.1469758436000@Atlassian.JIRA> <JIRA.12993427.1469758436300@arcas>
Subject: [jira] [Commented] (MAPREDUCE-6745) Job directories should be clean
 in staging directorg /tmp/hadoop-yarn/staging after MapReduce job finish
 successfully
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 03 Aug 2016 03:41:22 -0000


    [ https://issues.apache.org/jira/browse/MAPREDUCE-6745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15405252#comment-15405252 ] 

mujunchao commented on MAPREDUCE-6745:
--------------------------------------

We will move the .staging dir away while Job finished or failed. As the Job is not alive, i think no need to keep the .staging dir at that time.

> Job directories should be clean in staging directorg /tmp/hadoop-yarn/staging after MapReduce job finish successfully
> ---------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6745
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6745
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.7.2
>         Environment: Suse 11 sp3
>            Reporter: liuxiaoping
>            Priority: Blocker
>
> If MapReduce client set mapreduce.task.files.preserve.failedtasks=true, temporary job directory will not be deleted in staging directory /tmp/hadoop-yarn/staging.
> As time goes by, the job files are more and more, eventually lead to below exeception:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemExceededException):
> The directory item limit of /tmp/hadoop-yarn/staging/username/.staging is exceeded: limit=1048576 items=1048576
> 		at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:936)
> 		at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addLastINode(FSDirectory.java:981)
> 		at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.unprotectedMkdir(FSDirMkdirOp.java:237)
> 		at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createSingleDirectory(FSDirMkdirOp.java:191)
> 		at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createChildrenDirectories(FSDirMkdirOp.java:166)
> 		at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:97)
> 		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3788)
> 		at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:986)
> 		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:624)
> 		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolProtos.$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:624)
> 		at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> 		at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:973)
> 		at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2088)
> 		at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2084)
> 		at java.security.auth.Subject.doAs(Subject.java:422)
> 		at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1672)
> 		at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2082)
> 		
> 		
> The official instructions for the configuration mapreduce.task.files.preserve.failedtasks is below:
>     Should the files for failed tasks be kept. This should only be used on jobs that are failing, because the storage is never reclaimed. 
>     It also prevents the map outputs from being erased from the reduce directory as they are consumed.
> 	
> According to the instructions, I think the temporary files for successful tasks shouldn't be kept.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org