hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashwin Shankar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-4011) Jobs fail since nm-local-dir not cleaned up when rogue job fills up disk
Date Mon, 03 Aug 2015 19:04:04 GMT
Ashwin Shankar created YARN-4011:
------------------------------------

             Summary: Jobs fail since nm-local-dir not cleaned up when rogue job fills up
disk
                 Key: YARN-4011
                 URL: https://issues.apache.org/jira/browse/YARN-4011
             Project: Hadoop YARN
          Issue Type: Bug
          Components: yarn
    Affects Versions: 2.4.0
            Reporter: Ashwin Shankar


We observed jobs failed since tasks couldn't launch on nodes due to "java.io.IOException No
space left on device". 
On digging in further, we found a rogue job which filled up disk.
Specifically it was wrote a lot of map spills(like attempt_1432082376223_461647_m_000421_0_spill_10000.out)
to nm-local-dir causing disk to fill up, and it failed/got killed, but didn't clean up its
files in nm-local-dir.
So the disk remained full, causing subsequent jobs to fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message