hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Szita (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-7094) LocalDistributedCacheManager leaves classloaders open, which leaks FDs
Date Wed, 16 May 2018 16:11:00 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-7094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Adam Szita updated MAPREDUCE-7094:
----------------------------------
    Status: Patch Available  (was: In Progress)

> LocalDistributedCacheManager leaves classloaders open, which leaks FDs
> ----------------------------------------------------------------------
>
>                 Key: MAPREDUCE-7094
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7094
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Adam Szita
>            Assignee: Adam Szita
>            Priority: Major
>         Attachments: MAPREDUCE-7094.0.patch, MAPREDUCE-7094.1.patch, MAPREDUCE-7094.2.patch,
MAPREDUCE-7094.3.patch, MAPREDUCE-7094.4.patch, MAPREDUCE-7094.5.patch
>
>
> When a user starts a local mapred task from Hive's beeline, it will leave open file descriptors
on the HS2 process (which runs the mapred task).
> I debugged this and saw that it is caused by LocalDistributedCacheManager class, which
creates a new URLClassLoader, with a classpath for the two jars seen below. Somewhere down
the line Loaders will be created in this URLClassLoader for these files effectively creating
the FD's on the OS level.
> This is never cleaned up after execution, although LocalDistributedCacheManager removes
the files, it will not close the ClassLoader, so FDs are left open although they point to
deleted files at that time:
> {code:java}
> [root@host-1 ~]# lsof -p 14439 | grep hadoop-hive
> java    14439 hive  DEL       REG                8,1            
3348748 /tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar
> java    14439 hive  DEL       REG                8,1            
3348750 /tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar
> java    14439 hive  649r      REG                8,1   8112438   3348750
/tmp/hadoop-hive/mapred/local/1525789796609/hive-exec-1.1.0-cdh5.13.4-SNAPSHOT-core.jar (deleted)
> java    14439 hive  650r      REG                8,1   8112438   3348748
/tmp/hadoop-hive/mapred/local/1525789796610/hive-exec-core.jar (deleted)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message