ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Wagle (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-8074) Provide validated Oozie configs for Ambari deployments
Date Fri, 31 Oct 2014 18:37:34 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-8074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siddharth Wagle updated AMBARI-8074:
------------------------------------
    Description: 
Here are the list of changes that we need to have. Some are in place already - marked by *

* Have adminusers.txt with oozie install user *
* oozie.service.AuthorizationService.security.enabled = true *

For client retries during oozie server failure during restarts (during rolling upgrade for
example), oozie clients use a retry parameter.   The default is 4.   This is controlled by
the system property oozie.connection.retry.count.    The sleep time between retries is not
configurable and determined by 2^(retriynumber) seconds
So, so client will retry after 2, 4, 8 and 16 seconds delay.     If we need more delays than
the 30 seconds (cumulative) that we get now, we should set the system property (-Doozie.connection.retry.count=<value>)
in <oozie-insall-dir>/bin/oozie.distro file by setting OOZIE_CLIENT_OPTS env variable

We can also pass "-Doozie.connection.retry.count=<value>" on the command line when submitting
jobs.

- Sharelib handling

For rolling upgrade, we need to set these to be manually handled by the user - Even though
oozie guarantees to keep one older version, we would need to disable the automatic deletion
of older sharelibs.  The default is 1

oozie.service.ShareLibService.purge.interval=<numdays>  

Oozie sharelib is refreshed by explicit command or on oozie server restart.   We can't disable
the oozie-server restart refresh.   This can cause issues during rolling upgrade.   Today,
ambari installs the oozie sharelib as part of oozie install.   When multiple oozie servers
are involved, all oozie servers should be stopped before oozie upgrade is begun. And all oozie
server binaries and config should be updated and the sharelib updated just before the oozie
servers are  started with the new version bits

- Handling Hive/Tez. 
As it stands today, to ease the effort of users shipping hive and tez files, we should let
the installers create per action default config file prepopulated with the contents of hive
and tez (for hive actions) and pig and tez (for pig actions) so that workflows need not change
between rolling upgrades.   See BUG-26227 for more details on our attempts to fix it and why
this is necessary


  was:
Here are the list of changes that we need to have. Some are in place already - marked by *

* Have adminusers.txt with oozie install user *
* oozie.service.AuthorizationService.security.enabled = true *

For client retries during oozie server failure during restarts (during rolling upgrade for
example), oozie clients use a retry parameter.   The default is 4.   This is controlled by
the system property oozie.connection.retry.count.    The sleep time between retries is not
configurable and determined by 2^(retriynumber) seconds
So, so client will retry after 2, 4, 8 and 16 seconds delay.     If we need more delays than
the 30 seconds (cumulative) that we get now, we should set the system property (-Doozie.connection.retry.count=<value>)
in <oozie-insall-dir>/bin/oozie.distro file by setting OOZIE_CLIENT_OPTS env variable

We can also pass "-Doozie.connection.retry.count=<value>" on the command line when submitting
jobs.

- Sharelib handling

For rolling upgrade, we need to set these to be manually handled by the user - Even though
oozie guarantees to keep one older version, we would need to disable the automatic deletion
of older sharelibs.  The default is 1

oozie.service.ShareLibService.purge.interval=<numdays>  

- Oozie sharelib is refreshed by explicit command or on oozie server restart.   We can't disable
the oozie-server restart refresh.   This can cause issues during rolling upgrade.   Today,
ambari installs the oozie sharelib as part of oozie install.   When multiple oozie servers
are involved, all oozie servers should be stopped before oozie upgrade is begun. And all oozie
server binaries and config should be updated and the sharelib updated just before the oozie
servers are  started with the new version bits

- Handling Hive/Tez. 
As it stands today, to ease the effort of users shipping hive and tez files, we should let
the installers create per action default config file prepopulated with the contents of hive
and tez (for hive actions) and pig and tez (for pig actions) so that workflows need not change
between rolling upgrades.   See BUG-26227 for more details on our attempts to fix it and why
this is necessary



> Provide validated Oozie configs for Ambari deployments
> ------------------------------------------------------
>
>                 Key: AMBARI-8074
>                 URL: https://issues.apache.org/jira/browse/AMBARI-8074
>             Project: Ambari
>          Issue Type: Task
>          Components: ambari-server
>    Affects Versions: 1.7.0
>            Reporter: Siddharth Wagle
>            Assignee: Siddharth Wagle
>             Fix For: 1.7.0
>
>
> Here are the list of changes that we need to have. Some are in place already - marked
by *
> * Have adminusers.txt with oozie install user *
> * oozie.service.AuthorizationService.security.enabled = true *
> For client retries during oozie server failure during restarts (during rolling upgrade
for example), oozie clients use a retry parameter.   The default is 4.   This is controlled
by the system property oozie.connection.retry.count.    The sleep time between retries is
not configurable and determined by 2^(retriynumber) seconds
> So, so client will retry after 2, 4, 8 and 16 seconds delay.     If we need more delays
than the 30 seconds (cumulative) that we get now, we should set the system property (-Doozie.connection.retry.count=<value>)
in <oozie-insall-dir>/bin/oozie.distro file by setting OOZIE_CLIENT_OPTS env variable
> We can also pass "-Doozie.connection.retry.count=<value>" on the command line when
submitting jobs.
> - Sharelib handling
> For rolling upgrade, we need to set these to be manually handled by the user - Even though
oozie guarantees to keep one older version, we would need to disable the automatic deletion
of older sharelibs.  The default is 1
> oozie.service.ShareLibService.purge.interval=<numdays>  
> Oozie sharelib is refreshed by explicit command or on oozie server restart.   We can't
disable the oozie-server restart refresh.   This can cause issues during rolling upgrade.
  Today, ambari installs the oozie sharelib as part of oozie install.   When multiple oozie
servers are involved, all oozie servers should be stopped before oozie upgrade is begun. And
all oozie server binaries and config should be updated and the sharelib updated just before
the oozie servers are  started with the new version bits
> - Handling Hive/Tez. 
> As it stands today, to ease the effort of users shipping hive and tez files, we should
let the installers create per action default config file prepopulated with the contents of
hive and tez (for hive actions) and pig and tez (for pig actions) so that workflows need not
change between rolling upgrades.   See BUG-26227 for more details on our attempts to fix it
and why this is necessary



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message