ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dale Bradman <>
Subject Express Upgrade to HDP 2.5 fails on restarting the Hive Metastore
Date Thu, 20 Apr 2017 16:45:47 GMT

I'm on Ambari and HDP with a  view to doing an EXPRESS Upgrade to HDP-

It finishes the service checks and the starts restarting Hive where it fails on "Restarting
Hive Metastore"

The error is:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/",
line 259, in <module>
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/",
line 280, in execute
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/",
line 696, in restart
    self.pre_upgrade_restart(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/",
line 114, in pre_upgrade_restart
  File "/var/lib/ambari-agent/cache/common-services/HIVE/",
line 198, in upgrade_schema
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/",
line 208, in cached_kinit_executor
    if (now - datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f") > timedelta(minutes=expiration_time)):
  File "/usr/lib/python2.7/", line 325, in _strptime
    (data_string, format))
ValueError: time data '2016-12-19 22:37:33' does not match format '%Y-%m-%d %H:%M:%S.%f'

Doing some digging, the /usr/lib/python2.6/site-packages/resource_management/libraries/functions/
does not handle non millisecond timestamps (line 205).

The timestamp it is trying to analyse here is the /var/lib/ambari-agent/tmp/kinit_executor_cache/<<hash(hive.service.keytab|>>.
The timestamp within this file matches that of the error: "'2016-12-19 22:37:33'" - which
contains no miliseconds. There are other files in this directory all of which have timesstamps
with miliseconds, so I'm going to assume this is 1 in a million. Could you confirm this please?

Options for fixing in my opinion are:

1.     Adding an additional conditional statement before line 205 that determines if the timestamp
contains miliseconds

2.     Manually changing '2016-12-19 22:37:33' to '2016-12-19 22:37:33.000000'

This is an Express Upgrade, so downtime has been arranged but it's reaching critical point,
would option 2 work  for immediate impact?

I've also upgraded to Ambari on a test cluster and the code remains as it is.

Let me know if JIRA required.


View raw message