Return-Path: X-Original-To: apmail-ambari-user-archive@www.apache.org Delivered-To: apmail-ambari-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 172D718A40 for ; Wed, 27 Jan 2016 00:53:13 +0000 (UTC) Received: (qmail 72976 invoked by uid 500); 27 Jan 2016 00:53:12 -0000 Delivered-To: apmail-ambari-user-archive@ambari.apache.org Received: (qmail 72944 invoked by uid 500); 27 Jan 2016 00:53:12 -0000 Mailing-List: contact user-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ambari.apache.org Delivered-To: mailing list user@ambari.apache.org Received: (qmail 72925 invoked by uid 99); 27 Jan 2016 00:53:12 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jan 2016 00:53:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 593451A04F7; Wed, 27 Jan 2016 00:53:12 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.527 X-Spam-Level: *** X-Spam-Status: No, score=3.527 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HK_RANDOM_ENVFROM=0.626, HTML_MESSAGE=3, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id eFLyjaOuuIqG; Wed, 27 Jan 2016 00:53:00 +0000 (UTC) Received: from mail-lb0-f181.google.com (mail-lb0-f181.google.com [209.85.217.181]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 07E3A21392; Wed, 27 Jan 2016 00:52:57 +0000 (UTC) Received: by mail-lb0-f181.google.com with SMTP id cl12so101945327lbc.1; Tue, 26 Jan 2016 16:52:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=Z6Nu0gBhzuB7xWxQG4X1DPOyg+Lffr6OQpYrk2CdESI=; b=QetzNrVg9CyOITqg6pVG1X6gcBradCi2esXFG2yw8G2OAstQ/tHJ3EEgVkrMVKsZji fdDLQD5MHNly9pZ7MaeL5QuW7QTHb3L8YEUyz+elbwO62o4pr68Uuk+xRnM8HRtjkx8m vtsiZeVTGG9+bg7fpQB4sFACwG8ibb/+cmPWz2AgCtiX1IA1DyYZVDrhxVLCHBYV199/ MCZdzuzQNkmHobmRdANfbt9zCEBTqkrZ+1E/YJYWFChB3lhTbf0ehMoen5V4raVymWAp L8S2vAKRDGJdvF5/1HtZlUC3wVjG19z213S2MeEV6WrhMUwMMvDdFeSCqvPEVzKGnVJp wKdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=Z6Nu0gBhzuB7xWxQG4X1DPOyg+Lffr6OQpYrk2CdESI=; b=d1xGTGgoTKwlpu/tTDObUYtGte/M6pVTZup+baZXoeMhhQoVG6xo0JQFykos/lzNkV ELfaicssgmxdJgEoZ2GUfwHXUAgTQlaELyIeZjbkWEt4lNNUPu2DXhh6lWmpFFBnhCDz J2JrrBMehmkGDv7bh7HNAQRAjqGMWg+VGU4CPCh4JjWR0Vpss9R66uvQa45aG8wug/gl CfaxoLyqJIstjuLMxkOK/Hk0HBlFNkHHGaffx0LDIwPs0FMgdL1ae8zk68GoV95GSNW2 Z8qd3jZARTsPQHan8mFf4G+m3Lo145CjgPBbihY4S6eCRGTGzhtJ1DrpuDZGgJcP1dHO JJxw== X-Gm-Message-State: AG10YOTzcq22dgNcjV7yA5G0EC0+u/wA5A3wA/EPNaP075OVC6rmnIhU93+5JT8jI663OjgcSUu7JAJt5oZLxw== MIME-Version: 1.0 X-Received: by 10.112.149.2 with SMTP id tw2mr9998486lbb.75.1453855975433; Tue, 26 Jan 2016 16:52:55 -0800 (PST) Received: by 10.25.26.73 with HTTP; Tue, 26 Jan 2016 16:52:55 -0800 (PST) In-Reply-To: <1453834215226.42438@hortonworks.com> References: <1453834215226.42438@hortonworks.com> Date: Wed, 27 Jan 2016 08:52:55 +0800 Message-ID: Subject: Re: Is there any guide on how to write stackadvisor ? From: Jeff Zhang To: user@ambari.apache.org Cc: "dev@ambari.apache.org" Content-Type: multipart/alternative; boundary=047d7b342f4af86f1d052a4638bd --047d7b342f4af86f1d052a4638bd Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Mithun and Srimanth. What I want to get the number of node managers, the configured memory/cores of each node manager and the min/max allocation memory of yarn, and base on these information I will recommend configuration for spark-thrift-sparkconf.conf I will take a look the material you suggested. On Wed, Jan 27, 2016 at 2:50 AM, Srimanth Gunturi wrote: > Hi Jeff, > > The closest we have to documentation is the python doc comments we have i= n > the base abstract class of StackAdvisor - > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resour= ces/stacks/stack_advisor.py#L23 > > > Each stack advisor does 4 things for which there are 4 =E2=80=8Babstract = methods > > 1. recommend component layout - *def** recommendComponentLayout(self, > services, hosts)* > > 2. validate component layout - *def** validateComponentLayout(self, > services, hosts)* > > 3. recommend configurations based on layout - *def** > recommendConfigurations(self, services, hosts)* > > 4. validate configurations based on layout - *def** > validateConfigurations(self, services, hosts)* > > > If you notice, all 4 methods are passed 2 important pieces of information= : > > 1. services - Contains any/all information about services being installed > and which hosts they are installed on > > 2. hosts - Contains any/all information about hosts in cluster > > > The above 2 data structures will tell you anything about the cluster bein= g > created/edited. > > > So to the question about 'how to get the number of nodemangers' - it > should be available by parsing both these data structures. We have utilit= y > method called 'def getHostsWithComponent(self, serviceName, componentName= , > services, hosts)' which gets the hosts a particular component is > installed. You can see its usage at - > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resour= ces/stacks/HDP/2.0.6/services/stack_advisor.py#L722 > =E2=80=8B > > > Hoping this helps. > > Best regards, > > Srimanth > > > > > ------------------------------ > *From:* Mithun Mathew > *Sent:* Monday, January 25, 2016 8:50 PM > *To:* user@ambari.apache.org > *Cc:* dev@ambari.apache.org > *Subject:* Re: Is there any guide on how to write stackadvisor ? > > Hi Jeff > > Some existing implementation might help you give an idea of how > stack_advisor is implemented to component layout recommendation: > https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resour= ces/stacks/HDP/2.3/services/stack_advisor.py#L25-#L51 > > If you can explain what exactly you are trying to achieve, I might be abl= e > to help you with it. > > Regards > Matt > > On Mon, Jan 25, 2016 at 4:31 PM, Jeff Zhang wrote: > >> I try to add stackadvisor for one component. The logic is simple, but >> don't know how to do that use ambari api. e.g. how to get the number of >> nodemangers, how to get the number of hosts that with my service deploye= d. >> If there's any python api doc, there would be very helpful. >> >> >> -- >> Best Regards >> >> Jeff Zhang >> > > > > -- > *Mithun Mathew* (Matt) > > - www.linkedin.com/in/mithunmatt/ > > --=20 Best Regards Jeff Zhang --047d7b342f4af86f1d052a4638bd Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Mithun and Srimanth. What I want to get the number = of node managers, the configured memory/cores of each node manager and the = min/max allocation memory of yarn, and base on these information I will rec= ommend configuration for spark-thrift-sparkconf.conf

I w= ill take a look the material you suggested.=C2=A0

On Wed, Jan 27, 2016 at 2:50 AM= , Srimanth Gunturi <sgunturi@hortonworks.com> wrote:<= br>

Hi Jeff,

The closest we have to documentation is the python doc comments we have = in the=C2=A0base=C2=A0abstract class of StackAdvisor -=C2=A0https://github.com/apache/amba= ri/blob/trunk/ambari-server/src/main/resources/stacks/stack_advisor.py#L23<= /a>


Each stack advisor does 4 things for which there are 4 =E2=80=8Babstract= =C2=A0methods=C2=A0

1. recommend component layout -=C2=A0def recommendCom= ponentLayout(self, services, hosts)

2. validate component layout - de= f validateComponentL= ayout(self, services, hosts)

3. recommend configurations based on layout -=C2=A0def recommendConfigurations(self, services, hosts)

4. validate configurations based on layout -=C2=A0def= validateConfigurations(self, services, hosts)


If you notice, all 4 methods are passed 2 important pieces of informatio= n:

1. services - Contains any/all information about services being installe= d and which hosts they are installed on

2. hosts - Contains any/all information about hosts in cluster


The above 2 data structures will tell you anything about the cluster bei= ng created/edited.=C2=A0


So to the question about 'how to get the number of nodemangers' - it should be ava= ilable by parsing both these data structures. We have utility method called 'def getHostsWithComponen= t(self, serviceName, componentName, services, hosts)' which gets the hosts a particular component is installed. You can see its = usage at -=C2=A0https://github.com/apache/ambari/blob/trunk/ambari-ser= ver/src/main/resources/stacks/HDP/2.0.6/services/stack_advisor.py#L722= =E2=80=8B


Hoping this hel= ps.

Best regards,

Srimanth




From: Mithun Mathew <mithmatt@gmail.com>
Sent: Monday, January 25, 2016 8:50 PM
To: user= @ambari.apache.org
Cc: dev@a= mbari.apache.org
Subject: Re: Is there any guide on how to write stackadvisor ?
=C2=A0
Hi Jeff

Some existing implementation might help you give an idea of how stack_advis= or is implemented to component layout recommendation:=C2=A0https:/= /github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/stack= s/HDP/2.3/services/stack_advisor.py#L25-#L51

If you can explain what exactly you are trying to achieve, I might be able = to help you with it.

Regards
Matt

On Mon, Jan 25, 2016 at 4:31 PM, Jeff Zhang <zjffdu@gmail.com<= /a>> wrote:
I try to add stackadvisor for one component. The logic is simple, but = don't know how to do that use ambari api. e.g. how to get the number of= nodemangers, how to get the number of hosts that with my service deployed.= =C2=A0 If there's any python api doc, there would be very helpful.


--
Best Regards

Jeff Zhang



--



--
Best Regards

Jeff Zhang
--047d7b342f4af86f1d052a4638bd--