ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Hurley <jhur...@hortonworks.com>
Subject Re: Ru/EU not updating alert definitions in Ambari server db?
Date Tue, 23 Feb 2016 22:15:04 GMT
All good questions. I think that the RU/EU aspect of it can't really be a part of the discussion;
the orchestration done is really independent of the data stored in Ambari.  When using proper
inheritance, though, the example below will still work since the path still exists to the
original script. If the stacks have inheritance, it will be picked up. You can also define
the alerts anywhere within the stack. So if you needed to, you could store them in FLUME/alerts/alert_flume_whatever.py
in order to apply to any flume alert.

The upgrade catalogs do touch alert definitions on upgrade. In most cases, we edit the content;
in rare cases, we remove an outdated definition. So I think that this would normally be the
right place to perform the actions. Aside from that, if you're changing locations of the scripts,
I think the only course of action for alerts to also change their definitions. Ambari 2.4
is going to allow editing of script alerts via the UI - so that will certainly help for situations
like these.

> On Feb 23, 2016, at 4:17 PM, Di Li <lovelycoast@yahoo.com.INVALID> wrote:
> 
> Hello folks,
> 
> Yes, I would probably have had seen the error if I had looked at the UI (my bad, I didn't
check until post-RU). and yes the issue (since it's about refactored stack structure) will
be there the moment I upgrade the Ambari server.
> 
> I wanted to know if there is someway to have the alert definitions updated during ambari
server upgrade or RU/EU. I am keeping RU/EU in the loop because I may have different versions
of the same service in the common service mapped to service in stack. By that, I mean
>   common-services/FLUME/1.2.3.4.5 mapped to MyStack/1.0/FLUME
>   common-services/FLUME/1.3.4.5.1 mapped to MyStack/2.0/FLUMEwhere two versions of FLUME
have different alert_definition Python script.
> 
> In this case,I 'd rather have RU/EU take care of updating the alert definition when the
stack version is switched (from 1.0 to 2.0).
> 
> Understandably this may not be something currently in place in Ambari upgrade nor RU/EU,
so I figured I'd better check about this behavior on the mail list.
> Di
> 
>      From: Jonathan Hurley <jhurley@hortonworks.com>
> To: Ambari <dev@ambari.apache.org>; Di Li <lovelycoast@yahoo.com> 
> Sent: Tuesday, February 23, 2016 3:43 PM
> Subject: Re: Ru/EU not updating alert definitions in Ambari server db?
> 
> Based on this description, you should have seen the error before RU since you refactored
your stacks and moved the script. RU and, for that matter, Ambari upgrade, has no built-in
knowledge of your scripts. You can pretty much put them anywhere. So, if you change your directory
structure around, you're going to also have to update your definitions to have the right path.
> 
> This is why you should probably always put them in common services, and use inheritance
between stacks.
> 
>> On Feb 23, 2016, at 2:24 PM, Di Li <lovelycoast@yahoo.com.INVALID> wrote:
>> 
>> Hello folks,
>> 
>> Thanks for the reply. I have my own stack called MyStack 1.0 and MyStack 2.0. I found
the issue after I heavily refactored the MyStack directory structure.
>> 
>> 1. In Ambari 2.1/MyStack 1.0, I have Flume's alert_flume_agent_status.py defined
as MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py
>> 2. In Ambari 2.2, I changed the structure of MyStack to make use of common services,
so 
>>     a) The Flume directories are moved into common-services
>>     b) MyStack 1.0 now contains a Flume dir with just a metainfo.xml file.
>>     c) I also added MyStack 2.0 so I can run RU/EU. MyStack 2.0 also only contains
a Flume dir with just a metainfo.xml file.
>>     d) common-services has Flume agent alert Python script as FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py
(note the custom version I used here...). As you can see, I changed the path to the alert_flume_agent_status.py...
>> 3. Perform RU
>> 
>> -- Post RU, I saw Flume agent produced an "UNKNOWN" alert with an error message
>> Flume Agent Status
>> Unable to find 'MyStack/1.0/services/FLUME/package/alerts/alert_flume_agent_status.py'
as an absolute path or part of /var/lib/ambari-agent/cache/stacks or /var/lib/ambari-agent/cache/host_scripts
>> 
>> -- I noticed tht in the Ambari server database, the Flume agent alert definition
had the wrong patch to the Python script as shown below. It still has the old MyStack based
path.
>> 9 |          2 | flume_agent_status                            | FLUME          |
FLUME_HANDLER      |
>> ANY    | Flume Agent Status                              | This host-level alert
is triggered if any of the expect
>> ed flume agent processes are not available.
>>         |      1 |                1 | SCRIPT      | {"path":"MyStack/1.0/services/FLUME/package/alerts/alert_flu
>> me_agent_status.py","parameters":[{"name":"run.directory","display_name":"Run Directory","value":"/var/run/flume","de
>> scription":"The directory where flume agent processes will place their PID files.","type":"STRING"}],"type":"SCRIPT"}
>> 
>> -- This prompted me to ask the question about whether the alert definition should
have been updated by RU/EU (or ambari server upgrade maybe).
>> 
>> -- I was able to update Flume agent's alert definition with the correct path via
following REST API.
>> curl -u admin:admin -H 'X-Requested-By:ambari' -X PUT -d '{"AlertDefinition" : {"source"
: {"path" : "FLUME/1.2.3.4.5/package/alerts/alert_flume_agent_status.py",      "type" : "SCRIPT"
}}}' http://myhost.mydomain.com:8080/api/v1/clusters/test/alert_definitions/9
>> 
>> 
>>       From: Jonathan Hurley <jhurley@hortonworks.com>
>> To: "dev@ambari.apache.org" <dev@ambari.apache.org>; Di Li <lovelycoast@yahoo.com>

>> Sent: Tuesday, February 23, 2016 1:23 PM
>> Subject: Re: Ru/EU not updating alert definitions in Ambari server db?
>> 
>> No, they are not updated; however, stack inheritance should allow you to reference
the files from the new stack. We've performed many different upgrades both both inter-stack
and intra-stack and have never seen any issues with alert definitions. 
>> 
>>> On Feb 23, 2016, at 1:13 PM, Di Li <lovelycoast@yahoo.com.INVALID> wrote:
>>> 
>>> Hello folks,
>>> 
>>> Do RU and EU handle updating alert definitions stored in the Ambari server database?

>>> 
>>> Based on what I tested, alert definitions, especially the paths to alert_***.py
Python scripts are not updated by neither RU and EU. meaning if I changed the path to the
alert_***.py scripts in a new release, the alert definitions are broken and certain components
would show with a UNKNOWN alert state after I run RU/EU.
>>> 
>>> 
>>> P.S
>>>     I know that, post upgrade, I can manually update the alert definitions via
REST APIs.
>>> 
>>> Thanks.
>>> 
>>> 
>> 
>> 
> 
> 


Mime
View raw message