ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMBARI-20225) Take care of hive-interactive-site's 'hive.tez.container.size' config during HDP upgrade from 2.5 to 2.6 and update default value for tez-interactive-site's config 'tez.am.resource.memory.mb'.
Date Wed, 01 Mar 2017 21:10:45 GMT

    [ https://issues.apache.org/jira/browse/AMBARI-20225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891043#comment-15891043
] 

Hudson commented on AMBARI-20225:
---------------------------------

SUCCESS: Integrated in Jenkins build Ambari-branch-2.5 #1146 (See [https://builds.apache.org/job/Ambari-branch-2.5/1146/])
AMBARI-20225. Take care of hive-interactive-site's (sshridhar: [http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=ce1ed6e1aade5333c0ee26de469e8487061ccb8b])
* (edit) ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/tez-interactive-site.xml
* (edit) ambari-server/src/main/resources/stacks/HDP/2.5/services/HIVE/configuration/hive-interactive-site.xml
* (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml
* (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/config-upgrade.xml
* (edit) ambari-server/src/main/resources/stacks/HDP/2.5/services/stack_advisor.py
* (edit) ambari-server/src/main/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml
* (edit) ambari-server/src/test/python/stacks/2.5/common/test_stack_advisor.py


> Take care of hive-interactive-site's 'hive.tez.container.size' config during HDP upgrade
from 2.5 to 2.6 and update default value for tez-interactive-site's config 'tez.am.resource.memory.mb'.
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: AMBARI-20225
>                 URL: https://issues.apache.org/jira/browse/AMBARI-20225
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: trunk, 2.5.0
>            Reporter: Swapan Shridhar
>            Assignee: Swapan Shridhar
>             Fix For: trunk, 2.5.0
>
>         Attachments: AMBARI-20225.patch
>
>
> *Background :*
> We are dealing with 2 specific configs here. 
> *(1).* 'tez.am.resource.memory.mb' for tez-interactive-site and 
> *(2).* 'hive.tez.container.size' for hive-interactive-site. 
> - 'tez.am.resource.memory.mb' already exists in tez-interactive-site from Ambari-2.4
onwards and recently its default value had been changed to 'SET_ON_FIRST_INVOCATION' in its
hive-interacive-site.
> - 'hive.tez.container.size' was inherited and used from Hive1/hive-site in Ambari 2.4.
Starting from Ambari 2.5, it was added as a config explicitly in hive-interactive-site with
a default value of 'SET_ON_FIRST_INVOCATION'.
> The non-standard 'SET_ON_FIRST_INVOCATION' was introduced, so as to give a hint in Stack
Advisor (SA) call for the 1st time on Hive Server Interactive ON (enable_hive_interactive
config), so that config values for both can be calculated and set for the 1st time. Subsequesnt
calls in the lifecycle, will just read the value set by SA or if later changed by user manually.
> *Problem :*
> 'SET_ON_FIRST_INVOCATION', being non-standard was breaking the code in upgrade scenario.

> *Fix:*
> Removing the use of 'SET_ON_FIRST_INVOCATION' and adding code for correct behavior of
configs during HDP upgrade.
>  
> *Possible scenarios:*
> *1. Ambari upgrade from 2.4 - > 2.5 having HDP 2.5. Implying 2.5/stack_advisor.py
code would have been updated with new LLAP GA logic.*
>    - 'tez.am.resource.memory.mb' will already be there whether Hive Server Interactive
(HSI) is ON or OFF after Ambari upgrade. 
>       - If HSI if OFF before the upgrade was done, nothing to do over there.
>       - If HSI on ON before AMbari upgrade, or is made ON after Ambari upgrade, SA logic
has been updated to take of reading 'hive.tez.container.size' from hive-site as it wont exist
in hive-interactive-site at this point. It will show up in hive-interative-site aftre the
HDP upgrade to 2.6
> *2. Ambari 2.4 having HDP 2.5 -> Upgrade to Ambari 2.5 -> Upgraded to HDP 2.6*
>    - 'tez.am.resource.memory.mb' will already be there
>    - As part of HDP 2.5 upgrade to HDP 2.6, 'hive.tez.container.size' would have got
added, having current value picked from hive-site/hive.tez.container.size. SA code carries
a calculation logic for config 'hive.tez.container.size', which is bound to yield different
value compared to hive-site/'hive.tez.container.size'. The config's value will get updated/caluclated
(1). when HSI is made ON (meaning 'enabled_hive_interactive' config detected as **True** in
changed-configurations).    
>  
> *3. Ambari 2.5 with HDP 2.5 -> Upgraded to HDP 2.6*
>    - Both 'hive.tez.container.size' and 'tez.am.resource.memory.mb' (alreday present
at HDP 2.5 in hive-interactive-site) will exist after the HDP upgrade.
> *Note* that a fresh value for 'hive.tez.container.size' and 'tez.am.resource.memory.mb'
is caluclated at below conditions:
> (1). when HSI is made ON, meaning 'enabled_hive_interactive' config detected as **True**
in changed-configurations, or
> (2). it's a cluster create operation, where HSI also is supposed to be started. (BP scenario)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message