hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishna Kishore Bonagiri <write2kish...@gmail.com>
Subject Re: Extra start-up overhead with hadoop-2.1.0-beta
Date Fri, 02 Aug 2013 13:52:25 GMT
Hi Omkar,

  I have got these number by running a simple C program on the containers
that fetches the timestamp in microseconds and exits. The times mentioned
are low and high, they are not varying so drastically with in a version but
there are huge differences(like a second) between the two versions,
2.0.4-alpha and 2.1.0-beta as I mentioned.

  I am using a single node cluster, and all there is absolutely no other
load on the machine/node. My single node cluster is just used for my own
development work, and testing.

  I am not aware of what is resource localization, I am not doing anything
specially for that.

  Please let me know if you need any other info.


On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi <ojoshi@hortonworks.com> wrote:

> How are you making these measurements can you elaborate more? Is it on a
> best case basis or on an average or worst case? How many resources are you
> sending it for localization? were the sizes and number of these resources
> consistent across tests? Were these resources public/private/application
> specific? Apart from this is the other load on node manager same? is the
> load on hdfs same? did you see any network bottleneck?
> More information will help a lot.
> Thanks,
> Omkar Joshi
> *Hortonworks Inc.* <http://www.hortonworks.com>
> On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri <
> write2kishore@gmail.com> wrote:
>> Hi,
>>   Please share with me if you anyone has an answer or clues to my
>> question regarding the start up performance.
>> Also, one more thing I have observed today is the time taken to run a
>> command on a container went up by more than a second in this latest version.
>> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point
>> I call startContainer() to the  point the command is started on the
>> container.
>> where as
>> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it
>> came to the call back onContainerStarted() to the point the command is seen
>> started running on the container.
>> Thanks,
>> Kishore
>> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri <
>> write2kishore@gmail.com> wrote:
>>> Hi,
>>>   I have been using the hadoop-2.0.1-beta release candidate and observed
>>> that it is slower in running my simple application that runs on 2
>>> containers. I have tried to find out which parts of it is really having
>>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I
>>> found that.
>>> 1) From the point my Client has submitted the Application Master to RM,
>>> it is taking 2  seconds extra
>>> 2) From the point my container request are set up by Application Master,
>>> till the containers are allocated, it is taking 2 seconds extra
>>> Is this overhead expected with the changes that went into the new
>>> version? Or is there to improve it by changing something in configurations
>>> or so?
>>> Thanks,
>>> Kishore

View raw message