Return-Path: X-Original-To: apmail-ambari-user-archive@www.apache.org Delivered-To: apmail-ambari-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8EBCB179C4 for ; Sun, 10 May 2015 15:49:32 +0000 (UTC) Received: (qmail 87686 invoked by uid 500); 10 May 2015 15:49:32 -0000 Delivered-To: apmail-ambari-user-archive@ambari.apache.org Received: (qmail 87654 invoked by uid 500); 10 May 2015 15:49:32 -0000 Mailing-List: contact user-help@ambari.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ambari.apache.org Delivered-To: mailing list user@ambari.apache.org Received: (qmail 87644 invoked by uid 99); 10 May 2015 15:49:32 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 May 2015 15:49:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E740BC3C09 for ; Sun, 10 May 2015 15:49:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id BIzbAY0n9tpl for ; Sun, 10 May 2015 15:49:23 +0000 (UTC) Received: from mail-qg0-f43.google.com (mail-qg0-f43.google.com [209.85.192.43]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 4A49B215CA for ; Sun, 10 May 2015 15:49:22 +0000 (UTC) Received: by qgfi89 with SMTP id i89so57631115qgf.1 for ; Sun, 10 May 2015 08:49:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=uXNh+NcklgzHxGXV1BOtwWwnBv4ZTiQkGaMixZKDP2c=; b=PPspwmoxoo2iMsmM87rztp0loyjDRh7ErgECaYhskRD08wNF408cVL/9qLV8VXZQ+6 5jo27ydvRK6yHrZD3Uq1MzL6mytdANm0eHwdfi6zMv10/5BYwyvi+I18gpJLod3cZU7Q MHNql/XXCEEIt36VuD9/lXFTkUvPUB3wMYVAXqyaPwFNApGEZV8fWgICsqHXcu5hTfpP N3NT76HC3EPTaMtdputy85DgAmkeXrxfdShYz7dwoS/Jc4+FqHnT6i0N0WbB9VuoZX2f UMp5ekKErlmaoR/aKVMT/zxyqhAjBsTWl9jAsXV3/enMg9CFwXTVB/4dXPS2IvBsG/3Z XHnw== X-Received: by 10.55.23.16 with SMTP id i16mr14417733qkh.14.1431272954933; Sun, 10 May 2015 08:49:14 -0700 (PDT) Received: from oak.home (pool-173-62-185-209.phlapa.fios.verizon.net. [173.62.185.209]) by mx.google.com with ESMTPSA id t77sm8351186qgt.42.2015.05.10.08.49.12 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 10 May 2015 08:49:13 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_9E399FA7-C98D-419E-9CC9-810DB7BCB8EC" Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Subject: Re: Ambari Custom Service Questions. From: Christopher Jackson In-Reply-To: <1431221159223.50219@hortonworks.com> Date: Sun, 10 May 2015 11:49:11 -0400 Cc: Sumit Mohanty , afernandez@hortonworks.com Message-Id: <9F2B9E1F-1B00-47CB-AE10-671F9F15CA42@gmail.com> References: <747B6FB1-C7A4-4C4C-B885-05468EDDCAB0@gmail.com> <, <>> <1431221159223.50219@hortonworks.com> To: "user@ambari.apache.org" X-Mailer: Apple Mail (2.1993) --Apple-Mail=_9E399FA7-C98D-419E-9CC9-810DB7BCB8EC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Thanks for this information. I have a few follow up questions asked = inline. Thanks. Regards, Christopher Jackson > On May 9, 2015, at 9:25 PM, Sumit Mohanty = wrote: >=20 > One addition to question 4. > From: Alejandro Fernandez > > Sent: Saturday, May 09, 2015 1:44 PM > To: user@ambari.apache.org ; = Christopher Jackson > Subject: Re: Ambari Custom Service Questions. > =20 > Hi Christopher, these are all very good questions, and it would be = useful to supplement the wiki with them. > Comments inline. >=20 > On 5/9/15, 11:59 AM, "Christopher Jackson" = > wrote: >=20 > Hi All, >=20 > I=92ve been in the process of creating a custom Ambari service over = the past week and have quite a few general questions in which I haven=92t = found answers for in documentation or on the wiki. I was hoping some of = you could help answer any of the following questions. Thanks in advance. >=20 > 1) I=92ve noticed that when restarting a services component that is of = type =91CLIENT=92 that its install and configure method are invoked. I=92m= wondering if it=92s in intended and if so why? For components of type = =91MASTER=92 a restart doesn=92t seem to invoke install and configure = again, it just invokes stop then start. I ask about this because in my = custom service I have a CLIENT component in which there are some steps I = do in the install stage that I don=92t want repeated every time its = restarted. >=20 > Alejandro> For clients, a "Restart" or "Refresh Configs" essentially = only needs to make sure that the client libraries are present and the = configs are setup. Since technically a client cannot be restarted = because it is not a daemon, the code is written in such a way that it is = idempotent, so no harm in installing libs that are already present, or = settings configs that are already there. > For Masters, they have independent commands to Install the libs, and = Restart the daemon process. > For your client, is there any artifact you can check to avoid running = your one-time-install multiple times? Chris> Thank you for the explanation. Yes I can work around this by = checking if certain artifacts exist. >=20 > 2) Can someone explain the implication of the auto-deploy and its = child elements in the context where the following snippet would be = placed in the metainfo.xml file of a custom service component = (MY_SERVICE/MY_COMPONENT_MASTER)? >=20 > > HBASE/HBASE_CLIENT > host > > true > MY_SERVICE/MY_COMPONENT_CLIENT > > >=20 > Alejandro> This is a really good question. Some components depend on = others, and those dependencies need to be installed automatically, = either anywhere in the cluster, the same host, or on the same host that = contains another component. You probably don't need the "co-locate" tag, = since it is used to indicate masters that must be together, and is = configured in the UI during the Service Install Wizard. Chris> As part of my custom service I am adding libraries to both the = HDFS and HBASE Services. My custom services Client component ensures = these libraries get installed on the system and symlinked to the = appropriate lib folder. Here is my concern. If a user installs a cluster = with my service and then later adds an additional node to the cluster = with HDFS or HBASE installed on that node how can I ensure that my = custom services client component also gets installed? Is there a way to = do that without defining a custom stack or modifying the HBASE and HDFS = Service definitions? >=20 > 3) When creating a configuration file for a custom service what are = the valid entries for a tag? PASSWORD, and what else? Is = there any other child elements of =91property=92 that are useful? = Perhaps anything that allows you to provide a regular expression for = validation? >=20 > Alejandro> These are PASSWORD, USER, GROUP, TEXT. These only contain = name, value, description. If you need to use regex to validate a = property, that means the UI should take care of it, so take a look at = ambari-web module, particularly, config_property_helper.js >=20 > 4) Is there some function to restart a service in resource_management = or other ambari python library? Or should I just be restarting services = using the command line tools and ensuring to update the appropriate pid = files? I ask this question because I=92ve noticed I cannot restart a = service using the Ambari API as part of the installation/configuration = steps of my custom service, as the restart commands are queued while the = custom service installation/configuration is running, and will cause a = timeout. I=92m looking for a solution to this problem, what=92s = recommended if not one of the approaches I=92ve asked about above? >=20 > Alejandro> Once the service is defined in the metainfo.xml file, along = with the python file to use, it's up to that script to decide how to = install and restart your service. Ambari doesn't couple config changes = with forcing the service to restart automatically, this is because if a = user makes a config change or installs something, Ambari only highlights = that the service needs to be restarted, but it's up to the user to = decide when to do it. If you wanted to do automatic restarts upon config = changes, then that your python script would then have to call the = restart() method. Take a look at script.py >=20 > Sumit> In general the pattern of calling Ambari Server APIs from the = implementation of install/configure/start of a component definition is = not supported. This is because only one command can be executed at = anytime on a host. In theory, you could make the call from the = install/configure/start implementation and not wait for the call to = complete. Can you explain the scenario a bit more? Are you restarting = your custom service or some other service from the install/configuration = of the custom service.=20 Chris> As part of my custom service I alter configurations for HDFS, = HBASE, OOZIE, ZOOKEEPER, MAPREDUCE2, and YARN. I add libraries to HDFS, = HBASE, and OOZIE server (Symlink libraries from my component into the = specified services lib folder). To add the libraries to OOZIE server I=92m= stopping it, executing the oozie-setup shell script and then restarting = it. So to make sure these services pick up the new configurations, = additional libraries I wanted to restart them. Is there a better way to = handle adding the libraries for these services? >=20 > 5) How can I allow for the removal of a custom service from the ambari = console? I know there is a sequence of Ambari API commands I can run to: = stop the service, stop each of its components, and then finally delete = the service but is there any way to incorporate that into the Ambari UI = instead of having a script on the filesystem that invokes those calls? >=20 > Alejandro> Ambari UI allows deleting a component as long as it is = stopped. This is done from the Host Details page. But deleting a service = entirely is more complicated and I don't believe the API supports this = today (probably because it would have do delete records from a lot of = tables and is not a common use case). >=20 > Thanks so much, > Christopher Jackson >=20 >=20 --Apple-Mail=_9E399FA7-C98D-419E-9CC9-810DB7BCB8EC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Thanks for this information. I have a few follow up questions = asked inline. Thanks.

Regards,
Christopher Jackson

On May 9, 2015, at 9:25 PM, Sumit Mohanty <smohanty@hortonworks.com> wrote:

One addition to question 4.
From: Alejandro Fernandez <afernandez@hortonworks.com>
Sent: Saturday, May 09, 2015 1:44 = PM
To: user@ambari.apache.org; Christopher Jackson
Subject: Re: Ambari Custom Service = Questions.
 
Hi= Christopher, these are all very good questions, and it would be useful = to supplement the wiki with them.
Comments inline.

On 5/9/15, 11:59 AM, "Christopher Jackson" = <jackson.christopher.lee@gmail.com> wrote:

Hi All,

I=92ve been in the process of = creating a custom Ambari service over the past week and have quite a few = general questions in which I haven=92t found answers for in = documentation or on the wiki. I was hoping some of you could help answer = any of the following questions. Thanks in advance.

1) I=92ve noticed that when = restarting a services component that is of type =91CLIENT=92 that its = install and configure method are invoked. I=92m wondering if it=92s in = intended and if so why? For components of type =91MASTER=92 a restart = doesn=92t seem to invoke install and configure again, it just invokes = stop then start. I ask about this because in my custom service I have a = CLIENT component in which there are some steps I do in the install stage = that I don=92t want repeated every time its restarted.

Alejandro> For clients, a "Restart" or "Refresh Configs" = essentially only needs to make sure that the client libraries are = present and the configs are setup. Since technically a client cannot be = restarted because it is not a daemon, the code is written in such a way = that it is idempotent, so no harm in installing libs that are already = present, or settings configs that are already there.
For Masters, they have = independent commands to Install the libs, and Restart the daemon = process.
For your = client, is there any artifact you can check to avoid running your = one-time-install multiple = times?

Chris> Thank = you for the explanation. Yes I can work around this by checking if = certain artifacts exist.


2) Can someone explain the implication of = the auto-deploy and its child elements in the context where the = following snippet would be placed in the metainfo.xml file of a custom = service component (MY_SERVICE/MY_COMPONENT_MASTER)?

<dependency>
<name>HBASE/HBASE_CLIENT</name>
<scope>host</scope>
<auto-deploy>
<enabled>true</enabled>
<co-locate>MY_SERVICE/MY_COMPONENT_CLIENT</co-locate&g= t;
</auto-deploy>
</dependency>

Alejandro> This is a = really good question. Some components depend on others, and those = dependencies need to be installed automatically, either anywhere in the = cluster, the same host, or on the same host that contains another = component. You probably don't need the "co-locate" tag, since it is used = to indicate masters that must be together, and is configured in the UI = during the Service Install = Wizard.

Chris> As = part of my custom service I am adding libraries to both the HDFS and = HBASE Services. My custom services Client component ensures these = libraries get installed on the system and symlinked to the appropriate = lib folder. Here is my concern. If a user installs a cluster with my = service and then later adds an additional node to the cluster with HDFS = or HBASE installed on that node how can I ensure that my custom services = client component also gets installed? Is there a way to do that without = defining a custom stack or modifying the HBASE and HDFS Service = definitions?


3) When creating a = configuration file for a custom service what are the valid entries for a = <property-type> tag? PASSWORD, and what else? Is there any other = child elements of =91property=92 that are useful? Perhaps anything that = allows you to provide a regular expression for validation?

Alejandro> These are PASSWORD, USER, GROUP, TEXT. =  These only contain name, value, description. If you need to use = regex to validate a property, that means the UI should take care of it, = so take a look at ambari-web module, particularly, = config_property_helper.js

4) Is there some function to restart a = service in resource_management or other ambari python library? Or should = I just be restarting services using the command line tools and ensuring = to update the appropriate pid files? I ask this question because I=92ve = noticed I cannot restart a service using the Ambari API as part of the = installation/configuration steps of my custom service, as the restart = commands are queued while the custom service installation/configuration = is running, and will cause a timeout. I=92m looking for a solution to = this problem, what=92s recommended if not one of the approaches I=92ve = asked about above?

Alejandro> Once the = service is defined in the metainfo.xml file, along with the python file = to use, it's up to that script to decide how to install and restart your = service. Ambari doesn't couple config changes with forcing the service = to restart automatically, this is because if a user makes a config = change or installs something, Ambari only highlights that the service = needs to be restarted, but it's up to the user to decide when to do it. = If you wanted to do automatic restarts upon config changes, then that = your python script would then have to call the restart() method. Take a = look at script.py

Sumit> In general the pattern of calling Ambari Server = APIs from the implementation of install/configure/start of a component = definition is not supported. This is because only one command can be = executed at anytime on a host. In theory, you could make the = call from the install/configure/start implementation and = not wait for the call to complete. Can you explain the scenario a bit = more? Are you restarting your custom service or some other service from = the install/configuration of the custom = service. 

Chris> = As part of my custom service I alter configurations for HDFS, HBASE, = OOZIE, ZOOKEEPER, MAPREDUCE2, and YARN. I add libraries to HDFS, HBASE, = and OOZIE server (Symlink libraries from my component into the specified = services lib folder). To add the libraries to OOZIE server I=92m = stopping it, executing the oozie-setup shell script and then restarting = it. So to make sure these services pick up the new configurations, = additional libraries I wanted to restart them. Is there a better way to = handle adding the libraries for these services?


5) How can I allow for the removal of a = custom service from the ambari console? I know there is a sequence of = Ambari API commands I can run to: stop the service, stop each of its = components, and then finally delete the service but is there any way to = incorporate that into the Ambari UI instead of having a script on the = filesystem that invokes those calls?

Alejandro> Ambari UI allows = deleting a component as long as it is stopped. This is done from the = Host Details page. But deleting a service entirely is more complicated = and I don't believe the API supports this today = (probably because it would have do delete records from a lot of = tables and is not a common use case).

Thanks so much,
Christopher Jackson



= --Apple-Mail=_9E399FA7-C98D-419E-9CC9-810DB7BCB8EC--