hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4218) JobTracker may need to close its filesystem when being terminated
Date Fri, 19 Sep 2008 11:50:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632647#action_12632647
] 

Steve Loughran commented on HADOOP-4218:
----------------------------------------

stack traces of tests that fail once the JobTracker closes its filesystem when terminated

TestMultipleLevelCaching	testMultiLevelCaching	Error	Filesystem closed

java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:197)
at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:537)
at org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:201)
at org.apache.hadoop.mapred.TestMultipleLevelCaching.testCachingAtLevel(TestMultipleLevelCaching.java:116)
at org.apache.hadoop.mapred.TestMultipleLevelCaching.testMultiLevelCaching(TestMultipleLevelCaching.java:69)

TestRackAwareTaskPlacement	testTaskPlacement	Error	Filesystem closed

java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:197)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:574)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:400)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:651)
at org.apache.hadoop.mapred.TestRackAwareTaskPlacement.launchJobAndTestCounters(TestRackAwareTaskPlacement.java:78)
at org.apache.hadoop.mapred.TestRackAwareTaskPlacement.testTaskPlacement(TestRackAwareTaskPlacement.java:156)



> JobTracker may need to close its filesystem when being terminated
> -----------------------------------------------------------------
>
>                 Key: HADOOP-4218
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4218
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.19.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> This is something I've been experimenting with HADOOP-3268; I'm not sure what the right
action is here.
> -currently, the JobTracker does not close() its filesystem when it is shut down. This
will cause it to leak filesystem references if JobTrackers are started and stopped in the
same process.
> -The TestMRServerPorts test explicitly closes the filesystem
>         jt.fs.close();
>         jt.stopTracker();
> -If you move the close() operation into the stopTracker()/terminate logic, the filesystem
gets cleaned up, but 
> TestRackAwareTaskPlacement and TestMultipleLevelCaching fail with a FilesystemClosed
error (stack traces to follow)
> Should the JobTracker close its filesystem whenever it is terminated? If so, there are
some tests that need to be reworked slightly to not expect the fileystem to be live after
the jobtracker is taken down.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message