hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishna Kishore Bonagiri <write2kish...@gmail.com>
Subject Re: getAllocatedContainers() is not returning when ran against 2.0.3-alpha
Date Thu, 04 Apr 2013 06:14:57 GMT
Hi Hitesh,

  Thanks for the reply. I understand that it may not always allocate all
the requested containers in one go. After the actual request is sent, my
code repeatedly sends an empty container request and checks the responses
until all the requested containers are allocated.

  But my main worry is, even though the RM's log is saying that all 10
requested containers are allocated,  the getAllocatedContainers() method is
not returning me all of them, it returned only 9 surprisingly.

  I shall file a jira for this, but please let me know soon if you  have
any clue about this.


On Wed, Apr 3, 2013 at 9:20 PM, Hitesh Shah <hitesh@hortonworks.com> wrote:

> If I understand your question, you are expecting all the containers to be
> allocated in one go? Or are you seeing your application hang because it
> asked for 10 containers but it only received a total of 9 even after
> repeated calls to the RM?
> There is no guarantee that you will be allocated all your requested
> containers in a single allocate() call. The application is expected to keep
> on making allocate calls to the RM at regular intervals and the RM will
> give it any newly allocated containers in the period  since the last
> allocate call.
> The AMRMClient is  a library that should make it easier for someone to
> write an application master. Yes - it is the recommended approach. Please
> file jiras if you see use-cases that are not addressed by it when
> implementing your AM.
> thanks
> -- Hitesh
> On Apr 3, 2013, at 8:05 AM, Krishna Kishore Bonagiri wrote:
> > Hi,
> >
> >  I am running an application that had been written and working well with
> the hadoop-2.0.0-alpha but when I am running the same against 2.0.3-alpha,
> the getAllocatedContainers() method called on AMResponse is not returning
> all the containers allocated sometimes. For example, I request for 10
> containers and this method gives me only 9 containers sometimes, and when I
> looked at the log of Resource Manager, the 10th container is also
> allocated. It happens only sometimes randomly and works fine all other
> times. If I send one more request for the remaining container to RM, it
> could allocate that container. I am running only one application at a time,
> but 1000s of them one after another.
> >
> >   Is this problem seen by anyone else? I have seen that there are
> significant number of changes in the way ApplicationMaster.java is written
> now for the Distributed Shell example. There is now AMRMClient instead of
> AMRMProtocol. There is method addContainerRequest() which is not there
> earlier, and there is ConatainerRequest class now which was ResourceRequest
> earlier, etc.  Is it adivsable to re-write my application in a similar way?
> >
> > Thanks,
> > Kishore
> >
> >

View raw message