hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: FileSystem object's close method is not called. hadoop 2.7.2
Date Mon, 18 Jul 2016 16:55:53 GMT
Hello Michael,

Historically, there has never been a firm requirement that clients must call FileSystem#close
upon finishing usage of an instance.  I think the history here is that the close method was
not part of the initial API definition, and when it was added, there were already a lot of
existing applications coded that were not calling the close method.  A tactical decision was
made not to force all of those applications to make code changes, so calling close has been
treated as optional.  (This all pre-dates my time on the project though, so I might not have
the story completely right.)

I'd prefer a hard requirement that applications must call close, but this is our current situation.
 If you see a specific point in the codebase where you'd like to propose adding a close call,
then I recommend filing an Apache JIRA to discuss the proposal.  If you'd like, also consider
attaching a patch to the JIRA.

--Chris Nauroth

From: M G <mgbuystuff@gmail.com<mailto:mgbuystuff@gmail.com>>
Date: Wednesday, July 13, 2016 at 7:23 AM
To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>>
Subject: Fwd: FileSystem object's close method is not called. hadoop 2.7.2

Hi, I have implemented an HC file system by extending the org.apache.hadoop.fs.FileSystem
Things seem to work fine, except the fact that when I run mapreduce jobs with yarn (simple
DFSIO tests) I keep noticing that my Filesystem's "close" method is not called.
That is, I see in logs that my file system object is instantiated seven times during the run
of the MRAppMaster process, but the close method was never closed for any instantiation.
I know that there is a configuration key fs.automatic.close that is checked on MRappMaster.java
- and I make sure that this key is true.
I am using hadoop 2.7.2
Other processes that instantiate my FS class (node manager for example) are calling the close
method, I saw the problem only with MRAppMaster's process.
The DFSIO test succeeds, but this issue can lead to some resources leakage so it's important
that the close method is called.


View raw message