ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Swapan Shridhar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-16720) Update calculation logic for LLAP configs.
Date Wed, 18 May 2016 07:17:12 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-16720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Swapan Shridhar updated AMBARI-16720:
-------------------------------------
    Description: 
Following properties can/will get updated based on what triggered the calculation.

- LLAP concurrency - hive-interactive-site/hive.server2.tez.sessions.per.default.queue
- Number of LLAP nodes used - hive-interactive-env/num_llap_nodes
- LLAP YARN container Size (in MB) - hive-interactive-site/hive.llap.daemon.yarn.container.mb
- Number of Executors in container - hive-interactive-site/hive.llap.daemon.num.executors
- Cache Per node - hive-interactive-site/hive.llap.io.memory.size
- Property related to cache calculation above - hive-interactive-site/hive.llap.io.enabled
- LLAP Heap size per node - hive-interactive-env/llap_heap_size
- Slider App Master Container Size - hive-interactive-env/slider_am_container_size


The trigger point for updating LLAP configs (mentioned above) is change in values of any of
the following:

(1). 'enable_hive_interactive' set to 'true' 
(2). 'llap_queue_capacity' 
(3). 'hive.server2.tez.sessions.per.default.queue'
(4). 'llap' named queue getting selected for config 'hive.llap.daemon.queue.name' and only
2 queues exist ('llap' and 'default') at root level.

- If change in value for 'llap_queue_capacity' or 'hive.server2.tez.sessions.per.default.queue'
is detected, that config
value is not calulated, but read and use in calculation for dependent configs.

- If at any point of time, calulation cant be done (mostly because dependent config can't
be retrieved or LLAP queue capacity is not good enough), the value for configs are set to
their minimum or default values. A user can retrigger the calulation by sliding the "% of
cluster capacity" slider.


Brief on calculations is as follows :

- LLAP concurrency:

{code}
Calculated if this is not the driver for calculation, otherwise current value is read.

= 25% of LLAP queue size / 'Tez AM container Size' based on YARN min container size 
{code}

- Slider App Master Container Size:

{code}
= Calcuclated value lies b/w 256-1024 MB based on it's own value and YARN min container size.



total_am_capacity_required = normalized value of 'tez_am_container_size' based on YARN min
container size * llap_concurrency + Slider App Master Container Size
cap_available_for_daemons = total_llap_queue_size - total_am_capacity_required
{code}

- Number of LLAP nodes used:

{code}
= Normalized value of 'cap_available_for_daemons / yarn_nm_mem_in_mb' based on YARN min container
size.
{code}

- LLAP YARN container Size:

{code}
if Number of LLAP nodes used < 1.00: 
  = Normalized value of 'cap_available_for_daemons' based on YARN min container size.
else
  = Normalized value of 'yarn_nm_mem_in_mb' based on YARN min container size.
{code}

- Number of Executors in container:

{code}
= minimum (LLAP YARN container Size / hive_tez_container_size, Number of CPU's per Node Manager
Host).


total Mem for Executors = Number of Executors in container * Read value of hive_tez_container_size
{code}

- Cache Per node :

{code}
= LLAP YARN container Size - total Mem for Executors.

hive-interactive-site/hive.llap.io.enabled set to 'false' if 'Cache Per node' < 64, else
'true'.
{code}

- LLAP Heap size per node:

{code}
= maximum (total Mem for Executors * 0.8, total Mem for Executors - 1024)
{code}

  was:
Following properties can/will get updated based on what triggered the calculation.

- LLAP concurrency - hive-interactive-site/hive.server2.tez.sessions.per.default.queue
- Number of LLAP nodes used - hive-interactive-env/num_llap_nodes
- LLAP YARN container Size (in MB) - hive-interactive-site/hive.llap.daemon.yarn.container.mb
- Number of Executors in container - hive-interactive-site/hive.llap.daemon.num.executors
- Cache Per node - hive-interactive-site/hive.llap.io.memory.size
- Property related to cache calculation above - hive-interactive-site/hive.llap.io.enabled
- LLAP Heap size per node - hive-interactive-env/llap_heap_size
- Slider App Master Container Size - hive-interactive-env/slider_am_container_size


The trigger point for updating LLAP configs (mentioned above) is change in values of any of
the following:

(1). 'enable_hive_interactive' set to 'true' 
(2). 'llap_queue_capacity' 
(3). 'hive.server2.tez.sessions.per.default.queue'
(4). 'llap' named queue getting selected for config 'hive.llap.daemon.queue.name' and only
2 queues exist ('llap' and 'default') at root level.

- If change in value for 'llap_queue_capacity' or 'hive.server2.tez.sessions.per.default.queue'
is detected, that config
value is not calulated, but read and use in calculation for dependent configs.

- If at any point of time, calulation cant be done (mostly because dependent config can't
be retrieved or LLAP queue capacity is not good enough), the value for configs are set to
their minimum or default values. A user can retrigger the calulation by sliding the "% of
cluster capacity" slider.


Brief on calculations is as follows :

- LLAP concurrency:

Calculated if this is not the driver for calculation, otherwise current value is read.

= 25% of LLAP queue size / 'Tez AM container Size' based on YARN min container size 


- Slider App Master Container Size:

= Caluclated value lies b/w 256-1024 MB based on it's own value and YARN min container size.



total_am_capacity_required = normalized value of 'tez_am_container_size' based on YARN min
container size * llap_concurrency + Slider App Master Container Size
cap_available_for_daemons = total_llap_queue_size - total_am_capacity_required


Number of LLAP nodes used:

= Normalized value of 'cap_available_for_daemons / yarn_nm_mem_in_mb' based on YARN min container
size.


LLAP YARN container Size:

if Number of LLAP nodes used < 1.00: 
  = Normalized value of 'cap_available_for_daemons' based on YARN min container size.
else
  = Normalized value of 'yarn_nm_mem_in_mb' based on YARN min container size.


Number of Executors in container:

= minimum (LLAP YARN container Size / hive_tez_container_size, Number of CPU's per Node Manager
Host).


total Mem for Executors = Number of Executors in container * Read value of hive_tez_container_size


Cache Per node :

= LLAP YARN container Size - total Mem for Executors.

hive-interactive-site/hive.llap.io.enabled set to 'false' if 'Cache Per node' < 64, else
'true'.


LLAP Heap size per node:

= maximum (total Mem for Executors * 0.8, total Mem for Executors - 1024)


> Update calculation logic for LLAP configs.
> ------------------------------------------
>
>                 Key: AMBARI-16720
>                 URL: https://issues.apache.org/jira/browse/AMBARI-16720
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: 2.4.0
>            Reporter: Swapan Shridhar
>            Assignee: Swapan Shridhar
>             Fix For: 2.4.0
>
>         Attachments: AMBARI-16720.patch
>
>
> Following properties can/will get updated based on what triggered the calculation.
> - LLAP concurrency - hive-interactive-site/hive.server2.tez.sessions.per.default.queue
> - Number of LLAP nodes used - hive-interactive-env/num_llap_nodes
> - LLAP YARN container Size (in MB) - hive-interactive-site/hive.llap.daemon.yarn.container.mb
> - Number of Executors in container - hive-interactive-site/hive.llap.daemon.num.executors
> - Cache Per node - hive-interactive-site/hive.llap.io.memory.size
> - Property related to cache calculation above - hive-interactive-site/hive.llap.io.enabled
> - LLAP Heap size per node - hive-interactive-env/llap_heap_size
> - Slider App Master Container Size - hive-interactive-env/slider_am_container_size
> The trigger point for updating LLAP configs (mentioned above) is change in values of
any of the following:
> (1). 'enable_hive_interactive' set to 'true' 
> (2). 'llap_queue_capacity' 
> (3). 'hive.server2.tez.sessions.per.default.queue'
> (4). 'llap' named queue getting selected for config 'hive.llap.daemon.queue.name' and
only 2 queues exist ('llap' and 'default') at root level.
> - If change in value for 'llap_queue_capacity' or 'hive.server2.tez.sessions.per.default.queue'
is detected, that config
> value is not calulated, but read and use in calculation for dependent configs.
> - If at any point of time, calulation cant be done (mostly because dependent config can't
be retrieved or LLAP queue capacity is not good enough), the value for configs are set to
their minimum or default values. A user can retrigger the calulation by sliding the "% of
cluster capacity" slider.
> Brief on calculations is as follows :
> - LLAP concurrency:
> {code}
> Calculated if this is not the driver for calculation, otherwise current value is read.
> = 25% of LLAP queue size / 'Tez AM container Size' based on YARN min container size 
> {code}
> - Slider App Master Container Size:
> {code}
> = Calcuclated value lies b/w 256-1024 MB based on it's own value and YARN min container
size. 
> total_am_capacity_required = normalized value of 'tez_am_container_size' based on YARN
min container size * llap_concurrency + Slider App Master Container Size
> cap_available_for_daemons = total_llap_queue_size - total_am_capacity_required
> {code}
> - Number of LLAP nodes used:
> {code}
> = Normalized value of 'cap_available_for_daemons / yarn_nm_mem_in_mb' based on YARN min
container size.
> {code}
> - LLAP YARN container Size:
> {code}
> if Number of LLAP nodes used < 1.00: 
>   = Normalized value of 'cap_available_for_daemons' based on YARN min container size.
> else
>   = Normalized value of 'yarn_nm_mem_in_mb' based on YARN min container size.
> {code}
> - Number of Executors in container:
> {code}
> = minimum (LLAP YARN container Size / hive_tez_container_size, Number of CPU's per Node
Manager Host).
> total Mem for Executors = Number of Executors in container * Read value of hive_tez_container_size
> {code}
> - Cache Per node :
> {code}
> = LLAP YARN container Size - total Mem for Executors.
> hive-interactive-site/hive.llap.io.enabled set to 'false' if 'Cache Per node' < 64,
else 'true'.
> {code}
> - LLAP Heap size per node:
> {code}
> = maximum (total Mem for Executors * 0.8, total Mem for Executors - 1024)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message