ambari-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Madhuvanthi Radhakrishnan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-20880) Storm Log4J Regex Replacement on Upgrade Fails
Date Thu, 27 Apr 2017 20:30:04 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-20880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Madhuvanthi Radhakrishnan updated AMBARI-20880:
-----------------------------------------------
    Description: 
Parameterising log4j - AMBARI-19660 - Branch 2.5, trunk
<regex-replace> was added to all config-upgrade.xml starting from HDP2.3 upto HDP2.5
to change from log4j values to parameters like storm_log_maxbackupindex
In order to have a unique "find" key, above code uses
{noformat}
find : <RollingFile name="A1"
	*fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"*
		filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="100 MB"/>
replace : <RollingFile name="A1"
	fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"
	filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>
{noformat}
This logic works well for any upgrade from HDP2.5->HDP2.x
Because HDP2.5 inherits from Storm 1.0.1
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-worker-log4j.xml#L116
The above logic fails for all upgrades from HDP2.x to HDP2.5 because the configs are inherited
from Storm 0.10.0
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/storm-worker-log4j.xml#L115

https://issues.apache.org/jira/browse/AMBARI-19860 - log4j directory - trunk
The line change from 
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-worker-log4j.xml#L116
to 
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/storm-worker-log4j.xml#L115

was made in upgrade packs 2.3,2.4,2.5
This change is not required in upgrade pack 2.5 since it already inherits from Storm 1.0.1
Also, it was added under the storm_worker_log4j definition which is common for upgrade paths
2.3->2.4 as well which is not required. We want the log4j directory to change only for
any upgrades happening to 2.5

Fixes:
1. log4j parameterization
HDP Upgrade packs (config-upgrade) HDP2.3,HDP2.4 should refer to Storm 0.10.0 instead of Storm
1.0.1, for storm-worker-log4j, storm-cluster-log4j.
a. Change the find key to exclude the dependent line for storm-worker-log4j.
{noformat}
find : }.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="100 MB"/>
replace : }.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>
{noformat}
b. Modify find key to refer storm-cluster-log4j content from common-services/STORM/0.10.0
c. Better log message in https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigUpgradeChangeDefinition.java#L250

2. Resolve issues in https://hortonworks.jira.com/browse/BUG-67697
a. Change from regex-replace to replace
b. Remove the entry from HDP/2.5/config-upgrade. 
c. Create it as a separate entry in the config-upgrade files.
d. Call this entry only for nonrolling-upgrade-2.5.xml and upgrade-2.5.xml files.
e. Commit to 2.5.1

  was:
Parameterising log4j - AMBARI-19660 - Branch 2.5, trunk
<regex-replace> was added to all config-upgrade.xml starting from HDP2.3 upto HDP2.5
to change from log4j values to parameters like storm_log_maxbackupindex
In order to have a unique "find" key, above code uses
{noformat}
find : <RollingFile name="A1"
	*fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"*
		filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="100 MB"/>
replace : <RollingFile name="A1"
	fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"
	filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>
{noformat}
This logic works well for any upgrade from HDP2.5->HDP2.x
Because HDP2.5 inherits from Storm 1.0.1
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-worker-log4j.xml#L116
The above logic fails for all upgrades from HDP2.x to HDP2.5 because the configs are inherited
from Storm 0.10.0
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/storm-worker-log4j.xml#L115

https://issues.apache.org/jira/browse/AMBARI-19860 - log4j directory - trunk
The line change from 
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-worker-log4j.xml#L116
to 
https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/storm-worker-log4j.xml#L115

was made in upgrade packs 2.3,2.4,2.5
This change is not required in upgrade pack 2.5 since it already inherits from Storm 1.0.1
Also, it was added under the storm_worker_log4j definition which is common for upgrade paths
2.3->2.4 as well which is not required. We want to the log4j directory to change only for
any upgrades happening to 2.5

Fixes:
1. log4j parameterization
HDP Upgrade packs (config-upgrade) HDP2.3,HDP2.4 should refer to Storm 0.10.0 instead of Storm
1.0.1, for storm-worker-log4j, storm-cluster-log4j.
a. Change the find key to exclude the dependent line for storm-worker-log4j.
find : }.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="100 MB"/>
replace : }.%i.gz">
        <PatternLayout>
            <pattern>${pattern}</pattern>
        </PatternLayout>
        <Policies>
            <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>
b. Modify find key to refer storm-cluster-log4j content from common-services/STORM/0.10.0
c. Better log message in https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigUpgradeChangeDefinition.java#L250

2. Resolve issues in https://hortonworks.jira.com/browse/BUG-67697
a. Change from regex-replace to replace
b. Remove the entry from HDP/2.5/config-upgrade. 
c. Create it as a separate entry in the config-upgrade files.
d. Call this entry only for nonrolling-upgrade-2.5.xml and upgrade-2.5.xml files.
e. Commit to 2.5.1


> Storm Log4J Regex Replacement on Upgrade Fails
> ----------------------------------------------
>
>                 Key: AMBARI-20880
>                 URL: https://issues.apache.org/jira/browse/AMBARI-20880
>             Project: Ambari
>          Issue Type: Bug
>            Reporter: Madhuvanthi Radhakrishnan
>            Assignee: Madhuvanthi Radhakrishnan
>             Fix For: 2.5.1
>
>
> Parameterising log4j - AMBARI-19660 - Branch 2.5, trunk
> <regex-replace> was added to all config-upgrade.xml starting from HDP2.3 upto HDP2.5
to change from log4j values to parameters like storm_log_maxbackupindex
> In order to have a unique "find" key, above code uses
> {noformat}
> find : <RollingFile name="A1"
> 	*fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"*
> 		filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
>         <PatternLayout>
>             <pattern>${pattern}</pattern>
>         </PatternLayout>
>         <Policies>
>             <SizeBasedTriggeringPolicy size="100 MB"/>
> replace : <RollingFile name="A1"
> 	fileName="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}"
> 	filePattern="${sys:workers.artifacts}/${sys:storm.id}/${sys:worker.port}/${sys:logfile.name}.%i.gz">
>         <PatternLayout>
>             <pattern>${pattern}</pattern>
>         </PatternLayout>
>         <Policies>
>             <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>
> {noformat}
> This logic works well for any upgrade from HDP2.5->HDP2.x
> Because HDP2.5 inherits from Storm 1.0.1
> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-worker-log4j.xml#L116
> The above logic fails for all upgrades from HDP2.x to HDP2.5 because the configs are
inherited from Storm 0.10.0
> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/storm-worker-log4j.xml#L115
> https://issues.apache.org/jira/browse/AMBARI-19860 - log4j directory - trunk
> The line change from 
> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/1.0.1/configuration/storm-worker-log4j.xml#L116
> to 
> https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/resources/common-services/STORM/0.10.0/configuration/storm-worker-log4j.xml#L115

> was made in upgrade packs 2.3,2.4,2.5
> This change is not required in upgrade pack 2.5 since it already inherits from Storm
1.0.1
> Also, it was added under the storm_worker_log4j definition which is common for upgrade
paths 2.3->2.4 as well which is not required. We want the log4j directory to change only
for any upgrades happening to 2.5
> Fixes:
> 1. log4j parameterization
> HDP Upgrade packs (config-upgrade) HDP2.3,HDP2.4 should refer to Storm 0.10.0 instead
of Storm 1.0.1, for storm-worker-log4j, storm-cluster-log4j.
> a. Change the find key to exclude the dependent line for storm-worker-log4j.
> {noformat}
> find : }.%i.gz">
>         <PatternLayout>
>             <pattern>${pattern}</pattern>
>         </PatternLayout>
>         <Policies>
>             <SizeBasedTriggeringPolicy size="100 MB"/>
> replace : }.%i.gz">
>         <PatternLayout>
>             <pattern>${pattern}</pattern>
>         </PatternLayout>
>         <Policies>
>             <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>
> {noformat}
> b. Modify find key to refer storm-cluster-log4j content from common-services/STORM/0.10.0
> c. Better log message in https://github.com/apache/ambari/blob/trunk/ambari-server/src/main/java/org/apache/ambari/server/state/stack/upgrade/ConfigUpgradeChangeDefinition.java#L250
> 2. Resolve issues in https://hortonworks.jira.com/browse/BUG-67697
> a. Change from regex-replace to replace
> b. Remove the entry from HDP/2.5/config-upgrade. 
> c. Create it as a separate entry in the config-upgrade files.
> d. Call this entry only for nonrolling-upgrade-2.5.xml and upgrade-2.5.xml files.
> e. Commit to 2.5.1



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message