ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Swapan Shridhar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-20225) Take care of Hive2's 'hive.tez.container.size' config during HDP upgrade from 2.5 to 2.6
Date Tue, 28 Feb 2017 22:57:45 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Swapan Shridhar updated AMBARI-20225:
-------------------------------------
    Description: 
*Background :**

We are dealing with 2 specific configs here. 
*(1).* 'tez.am.resource.memory.mb' for tez-interactive-site and 
*(2).* 'hive.tez.container.size' for hive-interactive-site. 

- 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4 onwards
and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
- 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4. Starting
from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with a default
value of 'SET_ON_FIRST_INVOCATION'.

The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack Advisor
(SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive config),
so that config values for both can be calculated and set for the 1st time. Subsequesnt calls
in the lifecycle, will just read the value set by SA or if later changed by user manually.

**Problem :**
**---------**

'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario. 

**Fix:** 
**----**

Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of configs
during HDP upgrade.
 
**Possible scenarios:**
**------------------**

**1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py code
would have been updated with new LLAP GA logic.**
   - 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive (HSI)
is ON or OFF after Ambari upgrade. 
      - If HSI if OFF before the upgrade was done, nothing to do over there.
      - If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic has
been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist
in hive-interactive-site at this point. It will show up in hive-interative-site aftre the
HDP upgrade to 2.6


**2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6**
   - 'tez.am.resource.memory.mb' will already be there
   - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got added,
having current value picked from hive-site/hive.tez.container.size. SA code carries a calculation
logic for config 'hive.tez.container.size', which is bound to yield different value compared
to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated (1).
when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations).
   
 
**3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
   - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present at HDP
2.5 in hive-interactive-site) will exist after the HDP upgrade.


**Note** that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb'
is caluclated at below conditions:

(1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True** in
changed-configurations, or
(2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)

  was:
**Background :**
**----------** 

We are dealing with 2 specific configs here. 
**(1).** 'tez.am.resource.memory.mb' for tez-interactive-site and 
**(2).** 'hive.tez.container.size' for hive-interactive-site. 

- 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4 onwards
and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its hive-interacive-site.
- 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4. Starting
from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with a default
value of 'SET_ON_FIRST_INVOCATION'.

The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack Advisor
(SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive config),
so that config values for both can be calculated and set for the 1st time. Subsequesnt calls
in the lifecycle, will just read the value set by SA or if later changed by user manually.

**Problem :**
**---------**

'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario. 

**Fix:** 
**----**

Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of configs
during HDP upgrade.
 
**Possible scenarios:**
**------------------**

**1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py code
would have been updated with new LLAP GA logic.**
   - 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive (HSI)
is ON or OFF after Ambari upgrade. 
      - If HSI if OFF before the upgrade was done, nothing to do over there.
      - If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic has
been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist
in hive-interactive-site at this point. It will show up in hive-interative-site aftre the
HDP upgrade to 2.6


**2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6**
   - 'tez.am.resource.memory.mb' will already be there
   - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got added,
having current value picked from hive-site/hive.tez.container.size. SA code carries a calculation
logic for config 'hive.tez.container.size', which is bound to yield different value compared
to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated (1).
when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in changed-configurations).
   
 
**3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
   - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present at HDP
2.5 in hive-interactive-site) will exist after the HDP upgrade.


**Note** that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb'
is caluclated at below conditions:

(1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True** in
changed-configurations, or
(2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)


> Take care of Hive2's 'hive.tez.container.size' config during HDP upgrade from 2.5 to
2.6
> ----------------------------------------------------------------------------------------
>
>                 Key: AMBARI-20225
>                 URL: https://issues.apache.org/jira/browse/AMBARI-20225
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: 2.5.0
>            Reporter: Swapan Shridhar
>            Assignee: Swapan Shridhar
>             Fix For: 2.5.0
>
>
> *Background :**
> We are dealing with 2 specific configs here. 
> *(1).* 'tez.am.resource.memory.mb' for tez-interactive-site and 
> *(2).* 'hive.tez.container.size' for hive-interactive-site. 
> - 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4
onwards and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its
hive-interacive-site.
> - 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4.
Starting from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with
a default value of 'SET_ON_FIRST_INVOCATION'.
> The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack
Advisor (SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive
config), so that config values for both can be calculated and set for the 1st time. Subsequesnt
calls in the lifecycle, will just read the value set by SA or if later changed by user manually.
> **Problem :**
> **---------**
> 'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario.

> **Fix:** 
> **----**
> Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of
configs during HDP upgrade.
>  
> **Possible scenarios:**
> **------------------**
> **1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py
code would have been updated with new LLAP GA logic.**
>    - 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive
(HSI) is ON or OFF after Ambari upgrade. 
>       - If HSI if OFF before the upgrade was done, nothing to do over there.
>       - If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic
has been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist
in hive-interactive-site at this point. It will show up in hive-interative-site aftre the
HDP upgrade to 2.6
> **2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6**
>    - 'tez.am.resource.memory.mb' will already be there
>    - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got
added, having current value picked from hive-site/hive.tez.container.size. SA code carries
a calculation logic for config 'hive.tez.container.size', which is bound to yield different
value compared to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated
(1). when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in
changed-configurations).    
>  
> **3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6**
>    - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present
at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
> **Note** that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb'
is caluclated at below conditions:
> (1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True**
in changed-configurations, or
> (2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message