Return-Path: X-Original-To: apmail-ambari-user-archive@www.apache.org Delivered-To: apmail-ambari-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0BB2818A99 for ; Fri, 29 Jan 2016 05:39:55 +0000 (UTC) Received: (qmail 29590 invoked by uid 500); 29 Jan 2016 05:39:54 -0000 Delivered-To: apmail-ambari-user-archive@ambari.apache.org Received: (qmail 29558 invoked by uid 500); 29 Jan 2016 05:39:54 -0000 Mailing-List: contact user-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ambari.apache.org Delivered-To: mailing list user@ambari.apache.org Received: (qmail 29505 invoked by uid 99); 29 Jan 2016 05:39:54 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jan 2016 05:39:54 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 555CBC0D97; Fri, 29 Jan 2016 05:39:54 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.526 X-Spam-Level: *** X-Spam-Status: No, score=3.526 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HK_RANDOM_ENVFROM=0.626, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id UCTA3QVmAo3W; Fri, 29 Jan 2016 05:39:41 +0000 (UTC) Received: from mail-lf0-f44.google.com (mail-lf0-f44.google.com [209.85.215.44]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 0571343ECF; Fri, 29 Jan 2016 05:39:41 +0000 (UTC) Received: by mail-lf0-f44.google.com with SMTP id c192so40504457lfe.2; Thu, 28 Jan 2016 21:39:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=P179j1/ww/cqP7u5AYuuF48a0q0tlBQns/4VV5UHV84=; b=RnFFLsxYhSIbfCe92kf0nEQGRwx39/kTqiUVXmIPlSRIZZbmgrttGrW83EdOaWdVkb LswsGotcjN3cNE3OV5AWojcVBptpiS2FFxEgV8FjPEthPyKToFjZllw92ev1bR1tVCm1 I+j/nZUWP52ayfQcpmRHYgahbDQut00mF9/ioWIzzxirDpxzgKDEGkRCsyVGxPNkBhOB YU2CKII5CMEAv3s7IvEJxSzHLQvPzNXbUVpUPUp+qlz+Jnm14VtI44oj44L4p7pMUWIW R9tVMOwc/+fP14QNWDsxGuyBSUy7q+xaxQIvv7Z/n6Ht9CWvhDxIFLi5yamWMTnfjYCj +z6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=P179j1/ww/cqP7u5AYuuF48a0q0tlBQns/4VV5UHV84=; b=CY5KPQPLUfwN0txK3L626GVZ422FJRHve02qXv+y8wV8vj3oAUxv1pb7SCdKLRhifU SGGVmh/yFhMdkUpx5Q4ae0nRCR6yVL1ReFf8aOFQ6cY/45F9qzbDqNGNDnRVRARK9Tf0 IJfVP4T4v89HOUBIcpOrLsAW4c5dWszIYUW4dwIWIeTFzrdcbu3OS0XUOVfyD3KaNC7c 6+Zera7ULVihwNLk9+F7CkK0A0Wf5sIPwtDydt0CrLOYsSG5OT+tQTJT1g46mPvlabao JZda5K1HzEoIqRpzlmXL9huBtO8mqCcXM0RkNBmUk4OP4Y5BPiXBhNOsZJ/rgHX24qpv NfOA== X-Gm-Message-State: AG10YOQYjpCHDwqWWhpww+dbYxazciF3jI2Cn06DnX61i03CQxSG+Zq+x5MeG3KR1fy5TBw7M1g7SdtmTyih2A== MIME-Version: 1.0 X-Received: by 10.25.25.142 with SMTP id 136mr2670853lfz.42.1454045979953; Thu, 28 Jan 2016 21:39:39 -0800 (PST) Received: by 10.25.26.73 with HTTP; Thu, 28 Jan 2016 21:39:39 -0800 (PST) In-Reply-To: References: <1453834215226.42438@hortonworks.com> Date: Fri, 29 Jan 2016 13:39:39 +0800 Message-ID: Subject: Re: Is there any guide on how to write stackadvisor ? From: Jeff Zhang To: user@ambari.apache.org Cc: "dev@ambari.apache.org" Content-Type: multipart/alternative; boundary=001a11401e421f4f68052a7276d6 --001a11401e421f4f68052a7276d6 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks all, these are all very helpful. On Fri, Jan 29, 2016 at 10:15 AM, jun aoki wrote: > Hi Jeff, > > You may have already figured out, just another example here. > > https://github.com/Pivotal-Hadoop/ambari/blob/trunk/ambari-server/src/mai= n/resources/stacks/HDP/2.3/services/stack_advisor.py#L67 > (I believe) This counts how many HAWQ segments ( HAWQ's slave component) > there is in a cluster. > > On Tue, Jan 26, 2016 at 4:52 PM, Jeff Zhang wrote: > >> Thanks Mithun and Srimanth. What I want to get the number of node >> managers, the configured memory/cores of each node manager and the min/m= ax >> allocation memory of yarn, and base on these information I will recommen= d >> configuration for spark-thrift-sparkconf.conf >> >> I will take a look the material you suggested. >> >> On Wed, Jan 27, 2016 at 2:50 AM, Srimanth Gunturi < >> sgunturi@hortonworks.com> wrote: >> >>> Hi Jeff, >>> >>> The closest we have to documentation is the python doc comments we have >>> in the base abstract class of StackAdvisor - >>> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/reso= urces/stacks/stack_advisor.py#L23 >>> >>> >>> Each stack advisor does 4 things for which there are 4 =E2=80=8Babstrac= t methods >>> >>> 1. recommend component layout - *def** recommendComponentLayout(self, >>> services, hosts)* >>> >>> 2. validate component layout - *def** validateComponentLayout(self, >>> services, hosts)* >>> >>> 3. recommend configurations based on layout - *def** >>> recommendConfigurations(self, services, hosts)* >>> >>> 4. validate configurations based on layout - *def** >>> validateConfigurations(self, services, hosts)* >>> >>> >>> If you notice, all 4 methods are passed 2 important pieces of >>> information: >>> >>> 1. services - Contains any/all information about services being >>> installed and which hosts they are installed on >>> >>> 2. hosts - Contains any/all information about hosts in cluster >>> >>> >>> The above 2 data structures will tell you anything about the cluster >>> being created/edited. >>> >>> >>> So to the question about 'how to get the number of nodemangers' - it >>> should be available by parsing both these data structures. We have util= ity >>> method called 'def getHostsWithComponent(self, serviceName, >>> componentName, services, hosts)' which gets the hosts a particular >>> component is installed. You can see its usage at - >>> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/reso= urces/stacks/HDP/2.0.6/services/stack_advisor.py#L722 >>> =E2=80=8B >>> >>> >>> Hoping this helps. >>> >>> Best regards, >>> >>> Srimanth >>> >>> >>> >>> >>> ------------------------------ >>> *From:* Mithun Mathew >>> *Sent:* Monday, January 25, 2016 8:50 PM >>> *To:* user@ambari.apache.org >>> *Cc:* dev@ambari.apache.org >>> *Subject:* Re: Is there any guide on how to write stackadvisor ? >>> >>> Hi Jeff >>> >>> Some existing implementation might help you give an idea of how >>> stack_advisor is implemented to component layout recommendation: >>> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/reso= urces/stacks/HDP/2.3/services/stack_advisor.py#L25-#L51 >>> >>> If you can explain what exactly you are trying to achieve, I might be >>> able to help you with it. >>> >>> Regards >>> Matt >>> >>> On Mon, Jan 25, 2016 at 4:31 PM, Jeff Zhang wrote: >>> >>>> I try to add stackadvisor for one component. The logic is simple, but >>>> don't know how to do that use ambari api. e.g. how to get the number o= f >>>> nodemangers, how to get the number of hosts that with my service deplo= yed. >>>> If there's any python api doc, there would be very helpful. >>>> >>>> >>>> -- >>>> Best Regards >>>> >>>> Jeff Zhang >>>> >>> >>> >>> >>> -- >>> *Mithun Mathew* (Matt) >>> >>> - www.linkedin.com/in/mithunmatt/ >>> >>> >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > > -- > -jun > --=20 Best Regards Jeff Zhang --001a11401e421f4f68052a7276d6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks all, these are all very helpful.

On Fri, Jan 29, 2016 at 10:15 = AM, jun aoki <jaoki@apache.org> wrote:
Hi Jeff,=C2=A0

You may ha= ve already figured out, just another example here.=C2=A0
(I believe) This counts how many HAWQ segments ( HAWQ's slave componen= t) there is in a cluster.

On Tue, Jan 26, 2016 at 4:52 PM,= Jeff Zhang <zjffdu@gmail.com> wrote:
Thanks Mithun and Srimanth. What I want to get = the number of node managers, the configured memory/cores of each node manag= er and the min/max allocation memory of yarn, and base on these information= I will recommend configuration for spark-thrift-sparkconf.conf

I will take a look the material you suggested.=C2=A0

On Wed, Ja= n 27, 2016 at 2:50 AM, Srimanth Gunturi <sgunturi@hortonworks.com> wrote:

Hi Jeff,

The closest we have to documentation is the python doc comments we have = in the=C2=A0base=C2=A0abstract class of StackAdvisor -=C2=A0https://github.com/apache/amba= ri/blob/trunk/ambari-server/src/main/resources/stacks/stack_advisor.py#L23<= /a>


Each stack advisor does 4 things for which there are 4 =E2=80=8Babstract= =C2=A0methods=C2=A0

1. recommend component layout -=C2=A0def recommendCom= ponentLayout(self, services, hosts)

2. validate component layout - de= f validateComponentL= ayout(self, services, hosts)

3. recommend configurations based on layout -=C2=A0def recommendConfigurations(self, services, hosts)

4. validate configurations based on layout -=C2=A0def= validateConfigurations(self, services, hosts)


If you notice, all 4 methods are passed 2 important pieces of informatio= n:

1. services - Contains any/all information about services being installe= d and which hosts they are installed on

2. hosts - Contains any/all information about hosts in cluster


The above 2 data structures will tell you anything about the cluster bei= ng created/edited.=C2=A0


So to the question about 'how to get the number of nodemangers' - it should be ava= ilable by parsing both these data structures. We have utility method called 'def getHostsWithComponen= t(self, serviceName, componentName, services, hosts)' which gets the hosts a particular component is installed. You can see its = usage at -=C2=A0https://github.com/apache/ambari/blob/trunk/ambari-ser= ver/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py#L722= =E2=80=8B


Hoping this hel= ps.

Best regards,

Srimanth




From: Mithun Mathew <mithmatt@gmail.com>
Sent: Monday, January 25, 2016 8:50 PM
To: user= @ambari.apache.org
Cc: dev@a= mbari.apache.org
Subject: Re: Is there any guide on how to write stackadvisor ?
=C2=A0
Hi Jeff

Some existing implementation might help you give an idea of how stack_advis= or is implemented to component layout recommendation:=C2=A0https:/= /github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stack= s/HDP/2.3/services/stack_advisor.py#L25-#L51

If you can explain what exactly you are trying to achieve, I might be able = to help you with it.

Regards
Matt

On Mon, Jan 25, 2016 at 4:31 PM, Jeff Zhang <zjffdu@gmail.com<= /a>> wrote:
I try to add stackadvisor for one component. The logic is simple, but = don't know how to do that use ambari api. e.g. how to get the number of= nodemangers, how to get the number of hosts that with my service deployed.= =C2=A0 If there's any python api doc, there would be very helpful.


--
Best Regards

Jeff Zhang



--



--
Best Re= gards

Jeff Zhang



<= /div>--
-jun



--
Best Regards

Jeff Zhang
--001a11401e421f4f68052a7276d6--