hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3614) finalState UNDEFINED if AM is killed by hand
Date Sat, 25 Feb 2012 01:35:48 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216227#comment-13216227

Daryn Sharp commented on MAPREDUCE-3614:

Is there a compelling need to maintain a parallel set of JobIds?  Why not check the {{fileMap}}
values to see if the writer is open?  The event construction looks a bit odd, but I don't
know much about event injection.

I'd suggest reverting all the fs manipulation code, and try this:

-          new CompositeServiceShutdownHook(appMaster));
+          new CompositeServiceShutdownHook(appMaster) {
+            @Override
+            public void run() {
+              super.run();
+              try {
+                FileSystem.closeAll();
+              } catch (IOException e) {
+                // ignore?
+              }
+            }
+          });
       YarnConfiguration conf = new YarnConfiguration(new JobConf());
       conf.addResource(new Path(MRJobConfig.JOB_CONF_FILE));
       String jobUserName = System
       conf.set(MRJobConfig.USER_NAME, jobUserName);
+      conf.setBoolean("fs.automatic.close", false);
       initAndStartAppMaster(appMaster, conf, jobUserName);

Maybe you already tried this, but it prevents {{FileSystem}} from closing the fs objects automatically.
 Then you do the {{FS.closeAll()}} in your shutdown handler.
>  finalState UNDEFINED if AM is killed by hand
> ---------------------------------------------
>                 Key: MAPREDUCE-3614
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3614
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Ravi Prakash
>            Assignee: Ravi Prakash
>         Attachments: MAPREDUCE-3614.branch-0.23.patch, MAPREDUCE-3614.patch, MAPREDUCE-3614.patch,
MAPREDUCE-3614.patch, MAPREDUCE-3614.patch
> Courtesy [~dcapwell]
> {quote}
> If the AM is running and you kill the process (sudo kill #pid), the State in Yarn would
be FINISHED and FinalStatus is UNDEFINED.  The Tracking UI would say "History" and point to
the proxy url (which will redirect to the history server).
> The state should be more descriptive that the job failed and the tracker url shouldn't
point to the history server.
> {quote}

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message