hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prabhu Joseph <prabhujose.ga...@gmail.com>
Subject Re: YARN application container state doubt
Date Thu, 07 Jan 2016 11:12:33 GMT
Checking NMClientImpl.java, I think there is a possibility of containers
moving from ACQUIRED to COMPLETED state when there is any exception. By
default when AM MRAppMaster asks NM to start containers,
ContainerLauncherImpl has the logic to startContainers where i don't find
possibility of containers moving from ACQUIRED to COMPLETED state.  Want to
know what is the class NMClientImpl and where it is used in YARN.

Also the https://issues.apache.org/jira/browse/YARN-733 tells NMClientImpl
will return transit container status.

StartContainersResponse response
=proxy.getContainerManagementProtocol().startContainers(allRequests);
    startingContainer.state = ContainerState.RUNNING;
      } catch (YarnException e) {
        startingContainer.state = ContainerState.COMPLETE;
        removeStartedContainer(startingContainer);
        throw e;
      } catch (IOException e) {
        startingContainer.state = ContainerState.COMPLETE;
        removeStartedContainer(startingContainer);
        throw e;
      } catch (Throwable t) {
        startingContainer.state = ContainerState.COMPLETE;
        removeStartedContainer(startingContainer);
        throw RPCUtil.getRemoteException(t);
      }


Thanks,
Prabhu Joseph



On Thu, Jan 7, 2016 at 3:38 PM, Prabhu Joseph <prabhujose.gates@gmail.com>
wrote:

> Thanks Chris , that information was very useful. The code also shows
> ACQUIRED to COMPLETED is a valid state flow but could not able to find when
> that state change happens. Can someone help me here.
>
>
>
> On Thu, Jan 7, 2016 at 12:34 AM, Chris Nauroth <cnauroth@hortonworks.com>
> wrote:
>
>> If you have a copy of the Hadoop source and a working build environment,
>> then here is a cool way to answer questions about which state transitions
>> are valid.  There is a Maven build command that inspects the current
>> implementation of the state machines and summarizes the state transitions
>> in a GraphViz file.  There are visualization tools available for browsing
>> the GraphViz file, but they are plain text, so I tend to just browse them
>> in my text editor.
>>
>> This special build command is available for both ResourceManager and
>> NodeManager.  Your question relates to the ResourceManager, so let's look
>> at that one.
>>
>> > cd
>>
>> >hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-reso
>> >urcemanager/
>>
>> > mvn -o clean compile -Pvisualize
>>
>> > ls -lrt *.gv
>> -rw-r--r--+ 1 chris  staff    14K Jan  6 10:42 ResourceManager.gv
>>
>> Looking in that file, I can see that this is one of the state transitions
>> for RMContainer.
>>
>>
>>
>> subgraph cluster_RMContainer {
>> label="RMContainer"
>> ...
>> "RMContainer.ACQUIRED" -> "RMContainer.COMPLETED" [ label = "FINISHED" ];
>> ...
>>
>>
>> Based on this, we know that ACQUIRED to COMPLETED is a valid state
>> transition.  Unfortunately, I'm not deep enough on this code to quickly
>> answer the second part of your question: when would this happen?  Can
>> someone deeper on YARN please add to this?
>>
>>
>> --Chris Nauroth
>>
>>
>>
>>
>> On 1/6/16, 10:33 AM, "Prabhu Joseph" <prabhujose.gates@gmail.com> wrote:
>>
>> >Hi Experts,
>> >
>> >   I have a Custom ApplicationMaster written using TWILL which asks for
>> 20
>> >Containers and RM ApplicationMasterService has assigned 20 containers out
>> >of which 19 are moved from ACQUIRED to RUNNING. And there is one moved
>> >from
>> >*ACQUIRED to COMPLETED. *
>> >So my doubt is is ACQUIRED to COMPLETED a correct state flow and if so
>> >when
>> >that will happen.
>> >
>> >
>> >Thanks,
>> >Prabhu Joseph
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message