Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 235CA1020A for ; Tue, 26 Nov 2013 02:40:59 +0000 (UTC) Received: (qmail 98480 invoked by uid 500); 26 Nov 2013 02:40:53 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 98359 invoked by uid 500); 26 Nov 2013 02:40:53 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 98352 invoked by uid 99); 26 Nov 2013 02:40:53 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Nov 2013 02:40:53 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of write2kishore@gmail.com designates 209.85.128.48 as permitted sender) Received: from [209.85.128.48] (HELO mail-qe0-f48.google.com) (209.85.128.48) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 26 Nov 2013 02:40:49 +0000 Received: by mail-qe0-f48.google.com with SMTP id gc15so4957825qeb.7 for ; Mon, 25 Nov 2013 18:40:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=XqaExjw6qnq9sJ6N58KkEE7JPHXvOApvj5mCHkhhVsg=; b=d5rJlQSWmJloaj6v+Etlj0CIS9XmN7oovzfLT4ryYfncOA9qHwnK4NfZr0PIhOOIAr 3o+XU8EAUbGVYWPKWyFg2KSZK8U24a63oiBzu4hVj5vOxjB5ogsoPeXvIXER/6lCzpgV UC6wUK4j/CRFY3D1WrV+P0WbV056uEL3v9LrePK6jdv/+KZ8bVCJm06SLkq7pdqbHm0D Fxpy2aggN23a0MUfn4UaEugETtCd3A4TqIGGCKYTO2cIDEf08KSDMez4SasF3PgMF8rP 1J3w0lh+6z9sf+fPHvaJFnGL8NoDOy3Xeq0wfcXeNFt7dfDhW+5lthTngpRlT94iH1XI yxCw== MIME-Version: 1.0 X-Received: by 10.49.109.97 with SMTP id hr1mr52220194qeb.59.1385433628593; Mon, 25 Nov 2013 18:40:28 -0800 (PST) Received: by 10.96.215.100 with HTTP; Mon, 25 Nov 2013 18:40:28 -0800 (PST) In-Reply-To: References: <2075D494-C37A-4FC6-A509-356D05C11A22@hortonworks.com> Date: Tue, 26 Nov 2013 08:10:28 +0530 Message-ID: Subject: Re: Time taken for starting AMRMClientAsync From: Krishna Kishore Bonagiri To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7bea31a64ad41004ec0b678d X-Virus-Checked: Checked by ClamAV on apache.org --047d7bea31a64ad41004ec0b678d Content-Type: text/plain; charset=ISO-8859-1 Hi Alejandro, I don't start all the AMs from the same JVM. How can I do that? Also, when I do that, that will save me time taken to get AM started, which is also good to see an improvement in. Please let me know how can I do that? And, would this also save me time taken for connecting from AM to the Resource Manager? Thanks, Kishore On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur wrote: > Hi Krishna, > > Are you starting all AMs from the same JVM? Mind sharing the code you are > using for your time testing? > > Thx > > > On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore Bonagiri < > write2kishore@gmail.com> wrote: > >> Hi Alejandro, >> >> I have modified the code in >> >> >> hadoop-2.2.0-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/UnmanagedAMLauncher.java >> >> to submit multiple application masters one after another and still seeing >> 800 to 900 ms being taken for the start() call on AMRMClientAsync in all >> of those applications. >> >> Please suggest if you think I am missing something else >> >> Thanks, >> Kishore >> >> >> On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri < >> write2kishore@gmail.com> wrote: >> >>> Hi Alejandro, >>> >>> I don't know what are managed and unmanaged AMs, can you please >>> explain me what are the difference and how are each of them launched? >>> >>> I tried to google for these terms and came >>> across hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it >>> related to that? >>> >>> Thanks, >>> Kishore >>> >>> >>> On Tue, Nov 19, 2013 at 12:15 AM, Alejandro Abdelnur wrote: >>> >>>> Kishore, >>>> >>>> Also, please specify if you are using managed or unmanaged AMs (the >>>> numbers I've mentioned before are using unmanaged AMs). >>>> >>>> thx >>>> >>>> >>>> On Sun, Nov 17, 2013 at 11:16 AM, Vinod Kumar Vavilapalli < >>>> vinodkv@hortonworks.com> wrote: >>>> >>>>> It is just creating a connection to RM and shouldn't take that long. >>>>> Can you please file a ticket so that we can look at it? >>>>> >>>>> JVM class loading overhead is one possibility but 1 sec is a bit too >>>>> much. >>>>> >>>>> Thanks, >>>>> +Vinod >>>>> >>>>> On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote: >>>>> >>>>> Hi, >>>>> I am seeing the following call to start() on AMRMClientAsync taking >>>>> from 0.9 to 1 second. Why does it take that long? Is there a way to reduce >>>>> it, I mean does it depend on any of the interval parameters or so in >>>>> configuration files? I have tried reducing the value of the first argument >>>>> below from 1000 to 100 seconds also, but that doesn't help. >>>>> >>>>> AMRMClientAsync.CallbackHandler allocListener = new >>>>> RMCallbackHandler(); >>>>> amRMClient = AMRMClientAsync.createAMRMClientAsync(1000, >>>>> allocListener); >>>>> amRMClient.init(conf); >>>>> amRMClient.start(); >>>>> >>>>> >>>>> Thanks, >>>>> Kishore >>>>> >>>>> >>>>> >>>>> CONFIDENTIALITY NOTICE >>>>> NOTICE: This message is intended for the use of the individual or >>>>> entity to which it is addressed and may contain information that is >>>>> confidential, privileged and exempt from disclosure under applicable law. >>>>> If the reader of this message is not the intended recipient, you are hereby >>>>> notified that any printing, copying, dissemination, distribution, >>>>> disclosure or forwarding of this communication is strictly prohibited. If >>>>> you have received this communication in error, please contact the sender >>>>> immediately and delete it from your system. Thank You. >>>> >>>> >>>> >>>> >>>> -- >>>> Alejandro >>>> >>> >>> >> > > > -- > Alejandro > --047d7bea31a64ad41004ec0b678d Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Alejandro,

=A0 I don't start all the A= Ms from the same JVM. How can I do that? Also, when I do that, that will sa= ve me time taken to get AM started, which is also good to see an improvemen= t in. Please let me know how can I do that? And, would this also save me ti= me taken for connecting from AM to the Resource Manager?=A0

Thanks,
Kishore




On Tue, Nov 26, 2013 at 3:45 AM, Alejandro Abdelnur = <tucu@cloudera.co= m> wrote:
Hi Krishna,

<= div>Are you starting all AMs from the same JVM? Mind sharing the code you a= re using for your time testing?

Thx


On Thu, Nov 21, 2013 at 6:11 AM, Krishna Kishore= Bonagiri <write2kishore@gmail.com> wrote:
Hi Alejandro,

=A0I have modified the code in= =A0

hadoop-2.2.0-src/hadoop-yarn-project/hadoop-ya= rn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher/= src/main/java/org/apache/hadoop/yarn/applications/unmanagedamlauncher/Unman= agedAMLauncher.java

to submit multiple application masters one after = another and still seeing 800 to 900 ms being taken for the start() call on AMRMClientAsync= in all of those applications.

=
Please suggest if you think I am missing something else

Thanks,<= /span>
Kishore


On Tue, Nov 19, 2013 at 6:07 PM, Krishna Kishore Bonagiri <write2= kishore@gmail.com> wrote:
Hi Alejandro,

= =A0 I don't know what are managed and unmanaged AMs, can you please exp= lain me what are the difference and how are each of them launched?=A0

=A0I tried to google for these terms and came acr= oss=A0hadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar, is it relat= ed to that?

Thanks,
Kishore


On Tue, Nov 19, 2013 = at 12:15 AM, Alejandro Abdelnur <tucu@cloudera.com> wrote:
Kishore,

Also, please specify if you are using managed or unmanaged AMs (the number= s I've mentioned before are using unmanaged AMs).

thx


On Sun, Nov 17, 2013 at 11:16 AM, Vinod = Kumar Vavilapalli <vinodkv@hortonworks.com> wrote:
=
It is just creating a connection to RM = and shouldn't take that long. Can you please file a ticket so that we c= an look at it?

JVM class loading overhead is one possibi= lity but 1 sec is a bit too much.

Thanks,
+Vinod

On Oct 21, 2013, at 7:16 AM, Krishna Kishore Bonagiri wrote:<= /div>
Hi,
=A0 I am seeing= the following call to start() on AMRMClientAsync taking from 0.9 to 1 seco= nd. Why does it take that long? Is there a way to reduce it, I mean does it= depend on any of the interval parameters or so in configuration files? I h= ave tried reducing the value of the first argument below from 1000 to 100 s= econds also, but that doesn't help.

=A0 =A0 AMRMClientAsync.CallbackHandler allocListe= ner =3D new RMCallbackHandler();
=A0 =A0 amRMClient =3D AMRMClien= tAsync.createAMRMClientAsync(1000, allocListener);
=A0 =A0 amRMCl= ient.init(conf);
=A0 =A0 amRMClient.start();


<= div>Thanks,
Kishore



CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.



--
Alejandro





<= /div>--
Alejandro
--047d7bea31a64ad41004ec0b678d--