Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D98B510256 for ; Fri, 2 Aug 2013 13:53:03 +0000 (UTC) Received: (qmail 44488 invoked by uid 500); 2 Aug 2013 13:52:58 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 44073 invoked by uid 500); 2 Aug 2013 13:52:54 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 44058 invoked by uid 99); 2 Aug 2013 13:52:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Aug 2013 13:52:52 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of write2kishore@gmail.com designates 209.85.214.178 as permitted sender) Received: from [209.85.214.178] (HELO mail-ob0-f178.google.com) (209.85.214.178) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 02 Aug 2013 13:52:46 +0000 Received: by mail-ob0-f178.google.com with SMTP id ef5so1220109obb.9 for ; Fri, 02 Aug 2013 06:52:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=txxg8HOEEB+LCFDzrgZFDsaNCPU27ZRztOK7+UZZPy4=; b=ZkKWn/ZAAsshPN1zveGBbBIrfynqnnaYsmMyEju4/2l1tphzxBJop5lTREVW+dZXlx k5u7i1QtF7RQZMd9eF/q5CHoi89E3IJ5w6lRpESVjnsYJdwcSHhklHNnxyZARXTirMow iMzNLmVfIBOMf2jyyhzx0FCnw4mD0xxKB2leVDWBvuSULA8lbh5/22ZBXfvyN7IVCZIE Jd47VPvpuYEIVNjzkgTJnlFUdHRXJYHI6fNNE8tukXJX82UQ0iqjtK84MyYb8syaXhB3 Xm/++MslNZhKefbDO8sP9haw/ppe3ckrFdTp7cY7olQQ2kyu8tos97WKE+S03yFQZqNL O6oA== MIME-Version: 1.0 X-Received: by 10.42.33.129 with SMTP id i1mr615038icd.95.1375451545396; Fri, 02 Aug 2013 06:52:25 -0700 (PDT) Received: by 10.42.66.13 with HTTP; Fri, 2 Aug 2013 06:52:25 -0700 (PDT) In-Reply-To: References: Date: Fri, 2 Aug 2013 19:22:25 +0530 Message-ID: Subject: Re: Extra start-up overhead with hadoop-2.1.0-beta From: Krishna Kishore Bonagiri To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=bcaec51018f7c4fbea04e2f74485 X-Virus-Checked: Checked by ClamAV on apache.org --bcaec51018f7c4fbea04e2f74485 Content-Type: text/plain; charset=ISO-8859-1 Hi Omkar, I have got these number by running a simple C program on the containers that fetches the timestamp in microseconds and exits. The times mentioned are low and high, they are not varying so drastically with in a version but there are huge differences(like a second) between the two versions, 2.0.4-alpha and 2.1.0-beta as I mentioned. I am using a single node cluster, and all there is absolutely no other load on the machine/node. My single node cluster is just used for my own development work, and testing. I am not aware of what is resource localization, I am not doing anything specially for that. Please let me know if you need any other info. Thanks, Kishore On Thu, Aug 1, 2013 at 11:20 PM, Omkar Joshi wrote: > How are you making these measurements can you elaborate more? Is it on a > best case basis or on an average or worst case? How many resources are you > sending it for localization? were the sizes and number of these resources > consistent across tests? Were these resources public/private/application > specific? Apart from this is the other load on node manager same? is the > load on hdfs same? did you see any network bottleneck? > > More information will help a lot. > > > Thanks, > Omkar Joshi > *Hortonworks Inc.* > > > On Thu, Aug 1, 2013 at 2:19 AM, Krishna Kishore Bonagiri < > write2kishore@gmail.com> wrote: > >> Hi, >> Please share with me if you anyone has an answer or clues to my >> question regarding the start up performance. >> >> Also, one more thing I have observed today is the time taken to run a >> command on a container went up by more than a second in this latest version. >> >> When using 2.0.4-alpha, it used to take 0.3 to 0.5 seconds from the point >> I call startContainer() to the point the command is started on the >> container. >> >> where as >> >> When using 2.1.0-beta, it is taking around 1.5 seconds from the point it >> came to the call back onContainerStarted() to the point the command is seen >> started running on the container. >> >> Thanks, >> Kishore >> >> >> On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore Bonagiri < >> write2kishore@gmail.com> wrote: >> >>> Hi, >>> >>> I have been using the hadoop-2.0.1-beta release candidate and observed >>> that it is slower in running my simple application that runs on 2 >>> containers. I have tried to find out which parts of it is really having >>> this extra overhead(compared to hadoop-2.0.4-alpha), and here is what I >>> found that. >>> >>> 1) From the point my Client has submitted the Application Master to RM, >>> it is taking 2 seconds extra >>> 2) From the point my container request are set up by Application Master, >>> till the containers are allocated, it is taking 2 seconds extra >>> >>> Is this overhead expected with the changes that went into the new >>> version? Or is there to improve it by changing something in configurations >>> or so? >>> >>> Thanks, >>> Kishore >>> >> >> > --bcaec51018f7c4fbea04e2f74485 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Omkar,

=A0 I have got these number by runn= ing a simple C program on the containers that fetches the timestamp in micr= oseconds and exits. The times mentioned are low and high, they are not vary= ing so drastically with in a version but there are huge differences(like a = second) between the two versions, 2.0.4-alpha and 2.1.0-beta as I mentioned= .
=A0=A0
=A0 I am using a single node cluster, and all there i= s absolutely no other load on the machine/node. My single node cluster is j= ust used for my own development work, and testing.

=A0 I am not aware of what is resource localization, I am not doing anythi= ng specially for that.

=A0 Please let me know if you need any other info= .

Thanks,
Kishore


On Thu, Aug 1, 2013 at = 11:20 PM, Omkar Joshi <ojoshi@hortonworks.com> wrote:
How are you making these me= asurements can you elaborate more? Is it on a best case basis or on an aver= age or worst case? How many resources are you sending it for localization? = were the sizes and number of these resources consistent across tests? Were = these resources public/private/application specific? Apart from this is the= other load on node manager same? is the load on hdfs same? did you see any= network bottleneck?=A0

More information will help a lot.


Thanks,
Om= kar Joshi


On Thu, Aug 1, 2013 at 2:19 AM, Krishna = Kishore Bonagiri <write2kishore@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Hi,
=A0 Please share with me if you anyone has an answ= er or clues to my question regarding the start up performance.=A0

Also, one more thing I have observed today is the time take= n to run a command on a container went up by more than a second in this lat= est version.

When using 2.0.4-alpha, it used to take 0.3 to 0.5 seco= nds from the point I call startContainer() to the =A0point the command is s= tarted on the container.

where as

When using 2.1.0-beta, it is taking around 1.5 seconds from the = point it came to the call back=A0onContainerStarted() to the point the comm= and is seen started running on the container.

Than= ks,
Kishore


=
On Thu, Jul 25, 2013 at 8:38 PM, Krishna Kishore= Bonagiri <write2kishore@gmail.com> wrote:
Hi,

=A0 I have= been using the hadoop-2.0.1-beta release candidate and observed that it is= slower in running my simple application that runs on 2 containers. I have = tried to find out which parts of it is really having this extra overhead(co= mpared to hadoop-2.0.4-alpha), and here is what I found that.

1) From the point my Client has submitted the Applicati= on Master to RM, it is taking 2 =A0seconds extra
2) From the poin= t my container request are set up by Application Master, till the container= s are allocated, it is taking 2 seconds extra

Is this overhead expected with the changes that went in= to the new version? Or is there to improve it by changing something in conf= igurations or so?

Thanks,
Kishore



--bcaec51018f7c4fbea04e2f74485--