ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sumit Mohanty (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (AMBARI-14690) Configurable system resource values for ambari-agent
Date Fri, 22 Jan 2016 01:49:39 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-14690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111762#comment-15111762
] 

Sumit Mohanty edited comment on AMBARI-14690 at 1/22/16 1:48 AM:
-----------------------------------------------------------------

Hi [~oleewere] - Vinod, Sid and I had a discussion today. We came up with a solution which
needs to be provided in 2 phases - one short term and the second long term.

\\

Now let me provide a list of 3 different solutions (we have already discussed 1 and 2 above)
-
*Solution 1*: 
system resource properties in ambari-agent.ini (current patch you provided)
*Solution 2*: 
system resource properties in an external file, whose path is hard coded in ambari-agent.ini
*Solution 3*: 
system resource properties spread across multiple files located in an external directory (say
D1), whose path is hard coded in ambari-agent.ini. A property say _*system_resource_dir*_
could be added to the _*agent*_ section of ambari-agent.ini with value set to D1.

\\

A little bit of background: the resource-limits file that Yarn will generate, will be created
when the container comes up. Since ambari-agent.ini file will not exist at this point, the
current patch with solution (1) will not work. Now solution (2) will work, but we do not want
to restrict it to just 1 file. Hence we are proposing solution (3).

\\

In solution (3) all json files under D1 will be read by ambari agent and the properties will
override the default system values. If a property (say K1) is specified across 2 or more json
files under D1, then ambari agent should not bother about which value should prevail. Ambari
agent will read the files in any random order. The value of K1 will be set to whichever file
is read last. The onus is on the creator of files under D1 to ensure that property K1 is specified
in one and only one json file. In this case the onus will be on yarn.

\\

Phases of solution:
*Phase A - short term*: 
solution (3) above where the external directory (D1) is hard coded to this path - {{/yarn-private/ambari}}
*Phase B - long term*: 
solution (3) above where the external directory (D1) can be set to a configurable value from
the Ambari Server side

\\

Do you think you can provide Solution (3) as per Phase (A) in a reasonable time? We can have
a chat to clear any doubts on confusion on Friday Jan 22.



was (Author: gsaha):
Hi [~oleewere] - Vinod, Sid and I had a discussion today. We came up with a solution which
needs to be provided in 2 phases - one short term and the second long term.

\\

Now let me provide a list of 3 different solutions (we have already discussed 1 and 2 above)
-
*Solution 1*: 
system resource properties in ambari-agent.ini (current patch you provided)
*Solution 2*: 
system resource properties in an external file, whose path is hard coded in ambari-agent.ini
*Solution 3*: 
system resource properties spread across multiple files located in an external directory (say
D1), whose path is hard coded in ambari-agent.ini. A property say _*system_resource_dir*_
could be added to the _*agent*_ section of ambari-agent.ini with value set to D1.

\\

A little bit of background: the resource-limits file that Yarn will generate, will be created
when the container comes up. Since ambari-agent.ini file will not exist at this point, the
current patch with solution (1) will not work. Now solution (2) will work, but we do not want
to restrict it to just 1 file. Hence we are proposing solution (3).

\\

In solution (3) all json files under D1 will be read by ambari agent and the properties will
override the default system values. If a property (say K1) is specified across 2 or more json
files under D1, then ambari agent should not bother about which value should prevail. Ambari
agent will read the files in any random order. The value of K1 will be set to whichever file
is read last. The onus is on the creator of files under D1 to ensure that property K1 is specified
in one and only one json file. In this case the onus will be on yarn/ycloud.

\\

Phases of solution:
*Phase A - short term*: 
solution (3) above where the external directory (D1) is hard coded to this path - {{/yarn-private/ambari}}
*Phase B - long term*: 
solution (3) above where the external directory (D1) can be set to a configurable value from
the Ambari Server side

\\

Do you think you can provide Solution (3) as per Phase (A) in a reasonable time? We can have
a chat to clear any doubts on confusion on Friday Jan 22.


> Configurable system resource values for ambari-agent
> ----------------------------------------------------
>
>                 Key: AMBARI-14690
>                 URL: https://issues.apache.org/jira/browse/AMBARI-14690
>             Project: Ambari
>          Issue Type: Improvement
>          Components: ambari-agent
>    Affects Versions: 2.4.0
>            Reporter: Olivér Szabó
>            Assignee: Olivér Szabó
>             Fix For: 2.4.0
>
>         Attachments: AMBARI-14690.patch
>
>
> When an ambari agent starts, host system details are registered into ambari server database.
These values are calculated by ambari-agents based on different files on the hosts (e.g :
/proc/meminfo). 
> In some cases it isn't a correct behavior: If ambari-agent is in a docker-container,
it will see the same memory/cpu details. (stack advisor also uses these values, its possible
stack advisor can recommend a too high memory value for some services)
> Solution: Configurable system resources in ambari-agent.ini (if a value does not exist
or empty, it will use the default behavior)
> {code:java}
> [system_resource]
> processorcount=2
> physicalprocessorcount=2
> memorysize=50000
> memoryfree=50000
> memorytotal=100000
> swapsize=20
> swapfree=20
> ...
> {code}
> In ambari-agent Facter.py set these values, all of the factor info values can be redefined
in ambari-agent.ini file.
> That means ambari is not responsible to gather these values from the system.
> - use case: 
> During 'docker run' , if there is a script which calls 'ambari-agent start', before that,
some of the memory values can be replaced/inserted into the ambari-agent.ini file in that
script. (e.g. the user knows that, 4 ambari-agent containers will be started, with the same
services, so the real memory needs to be devided by 4 etc.)
> Facter.py is called during ambari-agent start, so if the values will be modified later,
ambari-agent needs to be restarted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message