hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-477) When default container executor fails right away, at the CLI launching our App Master, Client doesn't always get the signal to kill the job
Date Thu, 14 Mar 2013 23:44:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602892#comment-13602892
] 

Eli Reisman commented on YARN-477:
----------------------------------

Duh. Too many issues fixed all at once, they are all running together in my mind. OK, going
over this again, this is happening during my integration tests with MiniYARNCluster, not on
the real cluster.

So perhaps the real YARN implementation handles propagating the error to the client and RM
(etc) when the command line the client tries to use to launch the container for the AM fails.
I think its the MiniYARNCluster that is not handling this situation correctly.

Again, the issue is:

Client starts fine. Creates AMContainerSpec stuff and tries to request AM container. This
request includes the shell command to launch our AM in the container. Container shows up as
being granted and provisioned by RM, but from there the client hangs waiting for job success/fail,
saying it has "1 container used" the whole time (the AM failed container.) What seems to be
happening is this shell script fails in launching the AM in its container, so the container
just sits there forever. Lets check this in MiniYARNCluster and see.

I will try to "break" the Giraph MiniYARNCluster test again and recreate some decent log traces
leading up to the event and I will post here. Thanks!
                
> When default container executor fails right away, at the CLI launching our App Master,
Client doesn't always get the signal to kill the job
> -------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-477
>                 URL: https://issues.apache.org/jira/browse/YARN-477
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Eli Reisman
>            Assignee: Zhijie Shen
>
> I have been porting Giraph to YARN (GIRAPH-13 is the issue) and when I launch my App
Master, if the container command line runs it successfully, any failure in the App Master
or my launched Giraph Tasks promptly reports to Client and ends my job run. However, if the
command line sent to the app master container fails to launch it at all, the error exit code
is not propagating. My client hangs with the job at containersUsed == 1 and state == ACCEPTED
for as long as you want to sit and wait before CTRL-C'ing your way out.
> Disclaimer: this could be my fault. But I wanted to throw it out there in case its not.
I also (when this happens) not getting error logs since the app master never launched, so
I really have no visibility into why it failed to launch. I am sure its not launching, but
the client IS sending the app request, getting a container for my AM, and I see the command
line run on the container in my logs. Thats all.
> Thanks! If this is a dup or "won't fix" for some reason, let me know and sorry for wasting
your time!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message