hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9234) HiveServer2 leaks FileSystem objects in FileSystem.CACHE
Date Mon, 02 Feb 2015 06:09:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14300915#comment-14300915
] 

Thejas M Nair commented on HIVE-9234:
-------------------------------------

The leak was in FileSystem.CACHE, it would keep growing. The Key in the CACHE includes UGI
object. To clear the entries in CACHE corresponding to the UGI object, closeAllForUGI should
be called once the UGI is no longer going to be used. 
The fix is in changing the sequence - to do ShimLoader.getHadoopShims().closeAllForUGI(sessionUgi)
as the final step of HiveSessionImplwithUGI.close, after super.close().  super.close() was
indirectly adding another entry into the CACHE.


> HiveServer2 leaks FileSystem objects in FileSystem.CACHE
> --------------------------------------------------------
>
>                 Key: HIVE-9234
>                 URL: https://issues.apache.org/jira/browse/HIVE-9234
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 0.12.0, 0.13.0, 0.12.1, 0.14.0, 0.13.1
>            Reporter: Vaibhav Gumashta
>            Assignee: Vaibhav Gumashta
>             Fix For: 0.15.0, 1.0.0
>
>         Attachments: HIVE-9234.1.patch, HIVE-9234.2.patch, HIVE-9234.2.patch, HIVE-9234.branch-14.patch
>
>
> Running over extended period (48+ hrs), we've noticed HiveServer2 leaking FileSystem
objects in FileSystem.CACHE. Linked jiras were previous attempts to fix it, but the issue
still seems to be there. A workaround is to disable the caching (by setting {{fs.hdfs.impl.disable.cache}}
and {{fs.file.impl.disable.cache}} to {{true}}), but creating new FileSystem objects is expensive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message