libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gavin Williams (JIRA)" <j...@apache.org>
Subject [jira] [Created] (LIBCLOUD-1007) RFC3339 date parsing in GCE driver fails
Date Mon, 20 Aug 2018 11:16:00 GMT
Gavin Williams created LIBCLOUD-1007:
----------------------------------------

             Summary: RFC3339 date parsing in GCE driver fails
                 Key: LIBCLOUD-1007
                 URL: https://issues.apache.org/jira/browse/LIBCLOUD-1007
             Project: Libcloud
          Issue Type: Bug
          Components: Compute
            Reporter: Gavin Williams


When attempting to use the `deprecated` param on `GCENodeImage.deprecate()` [1], the call
was failing with:
{noformat}
2018-08-20 11:52:08 - DEBUG: Delete at = 2018-08-23T10:52:08.061384+00:00
Traceback (most recent call last):
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py",
line 6371, in ex_deprecate_image
timestamp_to_datetime(value)
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py",
line 57, in timestamp_to_datetime
ts = datetime.datetime.strptime(timestamp[:-10], '%Y-%m-%dT%H:%M:%S')
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py",
line 565, in _strptime_datetime
tt, fraction = _strptime(data_string, format)
File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/_strptime.py",
line 365, in _strptime
data_string[found.end():])
ValueError: unconverted data remains: .06

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "scripts/packer-image-manager.py", line 135, in <module>
main()
File "scripts/packer-image-manager.py", line 126, in main
deprecate_images()
File "scripts/packer-image-manager.py", line 56, in deprecate_images
dry_run=args.dry_run)
File "/Users/fatmcgav/Work/infra/infra/packer2.0/lib/utils.py", line 82, in deprecate_image
if image.deprecate(replacement_image, 'DEPRECATED', deleted=delete_at):
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py",
line 699, in deprecate
deprecated, obsolete, deleted)
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py",
line 6375, in ex_deprecate_image
attribute)
ValueError: deleted must be an RFC3339 timestamp
{noformat}
It would appear that the date parsing logic in `timestamp_to_datetime` [2] is flawed, assuming
that the length of `microseconds + tz offset` will always be 10 characters, whereas the above
example has 12 characters.

Patch applied as follows:
{code:java}
diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/gce.py
index 3dfc87d3..efcdcb87 100644
--- a/libcloud/compute/drivers/gce.py
+++ b/libcloud/compute/drivers/gce.py
@@ -52,9 +52,9 @@ def timestamp_to_datetime(timestamp):
:return: Datetime object corresponding to timestamp
:rtype: :class:`datetime.datetime`
"""
- # We remove timezone offset and microseconds (Python 2.5 strptime doesn't
- # support %f)
- ts = datetime.datetime.strptime(timestamp[:-10], '%Y-%m-%dT%H:%M:%S')
+ # We remove timezone offset
+ ts = datetime.datetime.strptime(timestamp[:timestamp.index('+')],
+ '%Y-%m-%dT%H:%M:%S.%f')
tz_hours = int(timestamp[-5:-3])
tz_mins = int(timestamp[-2:]) * int(timestamp[-6:-5] + '1')
tz_delta = datetime.timedelta(hours=tz_hours, minutes=tz_mins)
{code}
 

However once that issue was resolved, the Google Compute API call failed with:
{noformat}
2018-08-20 12:04:59 - DEBUG: Delete at = 2018-08-23T11:04:59.525758+00:00
Traceback (most recent call last):
File "scripts/packer-image-manager.py", line 135, in <module>
main()
File "scripts/packer-image-manager.py", line 126, in main
deprecate_images()
File "scripts/packer-image-manager.py", line 56, in deprecate_images
dry_run=args.dry_run)
File "/Users/fatmcgav/Work/infra/infra/packer2.0/lib/utils.py", line 82, in deprecate_image
if image.deprecate(replacement_image, 'DEPRECATED', deleted=delete_at):
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py",
line 699, in deprecate
deprecated, obsolete, deleted)
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py",
line 6381, in ex_deprecate_image
self.connection.request(request, method='POST', data=image_data).object
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/compute/drivers/gce.py",
line 124, in request
response = super(GCEConnection, self).request(*args, **kwargs)
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/common/google.py",
line 808, in request
*args, **kwargs)
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/common/base.py",
line 637, in request
response = responseCls(**kwargs)
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/common/base.py",
line 155, in __init__
self.object = self.parse_body()
File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-packages/libcloud/common/google.py",
line 288, in parse_body
raise InvalidRequestError(message, self.status, code)
libcloud.common.google.InvalidRequestError: {'domain': 'global', 'reason': 'invalid', 'message':
"Invalid value for field 'deprecationStatus.deleted': '2018-08-23T11:04:59.525758'. Must be
a string representation of a full-date or date-time in valid RFC3339 format with either 0
or 3 digits for fractional seconds"}
{noformat}
Tracked this down to the `ex_deprecate_image`  [3] call, which appeared to be 'validating'
the format of the timestamp, but not actually storing the converted result.

Another patch applied:
{code:java}
diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/gce.py
index 3dfc87d3..a3329133 100644
--- a/libcloud/compute/drivers/gce.py
+++ b/libcloud/compute/drivers/gce.py
@@ -6523,11 +6523,11 @@ class GCENodeDriver(NodeDriver):
continue

try:
- timestamp_to_datetime(value)
+ value = timestamp_to_datetime(value)
except:
raise ValueError('%s must be an RFC3339 timestamp' %
attribute)
- image_data[attribute] = value
+ image_data[attribute] = value.isoformat(timespec='milliseconds')

request = '/global/images/%s/deprecate' % (image.name)

{code}
Pull request incoming with both patches.

[1] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/gce.py#L6485-L6486]

[2] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/gce.py#L44-L61] 

[3] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drivers/gce.py#L6525-L6530] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message