ambari-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dale Bradman <da...@profusion.com>
Subject Express Upgrade to HDP 2.5 fails on restarting the Hive Metastore
Date Thu, 20 Apr 2017 16:45:47 GMT
Hi.

I'm on Ambari 2.4.2.0-136 and HDP 2.4.2.0 with a  view to doing an EXPRESS Upgrade to HDP-2.5.3.0.

It finishes the service checks and the starts restarting Hive where it fails on "Restarting
Hive Metastore"

The error is:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 259, in <module>
    HiveMetastore().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 280, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
line 696, in restart
    self.pre_upgrade_restart(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 114, in pre_upgrade_restart
    self.upgrade_schema(env)
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py",
line 198, in upgrade_schema
    status_params.tmp_dir)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py",
line 208, in cached_kinit_executor
    if (now - datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f") > timedelta(minutes=expiration_time)):
  File "/usr/lib/python2.7/_strptime.py", line 325, in _strptime
    (data_string, format))
ValueError: time data '2016-12-19 22:37:33' does not match format '%Y-%m-%d %H:%M:%S.%f'


Doing some digging, the /usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py
does not handle non millisecond timestamps (line 205).

The timestamp it is trying to analyse here is the /var/lib/ambari-agent/tmp/kinit_executor_cache/<<hash(hive.service.keytab|hive.principal.name)>>.
The timestamp within this file matches that of the error: "'2016-12-19 22:37:33'" - which
contains no miliseconds. There are other files in this directory all of which have timesstamps
with miliseconds, so I'm going to assume this is 1 in a million. Could you confirm this please?

Options for fixing in my opinion are:

1.     Adding an additional conditional statement before line 205 that determines if the timestamp
contains miliseconds

2.     Manually changing '2016-12-19 22:37:33' to '2016-12-19 22:37:33.000000'

This is an Express Upgrade, so downtime has been arranged but it's reaching critical point,
would option 2 work  for immediate impact?

I've also upgraded to Ambari 2.5.0.3-7 on a test cluster and the code remains as it is.

Let me know if JIRA required.

Cheers,
Dale


Mime
View raw message