From notifications-return-14704-archive-asf-public=cust-asf.ponee.io@libcloud.apache.org Mon Aug 20 13:30:06 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id BE190180663 for ; Mon, 20 Aug 2018 13:30:05 +0200 (CEST) Received: (qmail 51986 invoked by uid 500); 20 Aug 2018 11:30:04 -0000 Mailing-List: contact notifications-help@libcloud.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@libcloud.apache.org Delivered-To: mailing list notifications@libcloud.apache.org Received: (qmail 51977 invoked by uid 99); 20 Aug 2018 11:30:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Aug 2018 11:30:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 7EDBC1A14CD for ; Mon, 20 Aug 2018 11:30:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id a-JNrmbMS389 for ; Mon, 20 Aug 2018 11:30:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id E9FE55F439 for ; Mon, 20 Aug 2018 11:30:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 027B2E0DAA for ; Mon, 20 Aug 2018 11:30:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 4562D246A3 for ; Mon, 20 Aug 2018 11:30:00 +0000 (UTC) Date: Mon, 20 Aug 2018 11:30:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: notifications@libcloud.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (LIBCLOUD-1007) RFC3339 date parsing in GCE driver fails MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/LIBCLOUD-1007?page=3Dcom.atlass= ian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D1= 6585814#comment-16585814 ]=20 ASF GitHub Bot commented on LIBCLOUD-1007: ------------------------------------------ GitHub user fatmcgav opened a pull request: https://github.com/apache/libcloud/pull/1232 [LIBCLOUD-1007] Resolve RFC3339 date parsing issues with GCE Compute dr= iver. Resolves issue with RFC3339 date parsing in the GCE Compute driver.=20 Timestamp strings are now parsed upto and including the microsecond val= ue.=20 The call to deprecate the Google Compute image now also correctly forma= ts the date as RFC3339 format.=20 =20 Fixes https://issues.apache.org/jira/browse/LIBCLOUD-1007 =20 =20 - done, ready for review =20 =20 - [ ] [Code linting](http://libcloud.readthedocs.org/en/latest/developm= ent.html#code-style-guide) (required, can be done after the PR checks) - [ ] Documentation - [ ] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html) - [ ] [ICLA](http://libcloud.readthedocs.org/en/latest/development.html= #contributing-bigger-changes) (required for bigger changes) You can merge this pull request into a Git repository by running: $ git pull https://github.com/fatmcgav/libcloud gce_fix_rfc3339_parsing Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/1232.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1232 =20 ---- commit d3ae06dcb95725db7d6474f193e182e3e4aa46fc Author: Gavin Williams Date: 2018-08-20T11:20:22Z [gce] Fix parsing of timestamp string. Now parses microseconds as part of strptime function, stopping at the '= +' character to denote a timezone offset. commit 37a6d9adccbe0f6b1373708a4f7fcc14a38642b7 Author: Gavin Williams Date: 2018-08-20T11:21:31Z [gce] Use return value from `timestamp_to_datetime` and correctly forma= t the request datetime to RFC3339 format. ---- > RFC3339 date parsing in GCE driver fails > ---------------------------------------- > > Key: LIBCLOUD-1007 > URL: https://issues.apache.org/jira/browse/LIBCLOUD-1007 > Project: Libcloud > Issue Type: Bug > Components: Compute > Reporter: Gavin Williams > Priority: Major > > When attempting to use the `deprecated` param on `GCENodeImage.deprecate(= )` [1], the call was failing with: > {noformat} > 2018-08-20 11:52:08 - DEBUG: Delete at =3D 2018-08-23T10:52:08.061384+00:= 00 > Traceback (most recent call last): > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/compute/drivers/gce.py", line 6371, in ex_deprecate_image > timestamp_to_datetime(value) > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/compute/drivers/gce.py", line 57, in timestamp_to_datetime > ts =3D datetime.datetime.strptime(timestamp[:-10], '%Y-%m-%dT%H:%M:%S') > File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versio= ns/3.6/lib/python3.6/_strptime.py", line 565, in _strptime_datetime > tt, fraction =3D _strptime(data_string, format) > File "/usr/local/Cellar/python/3.6.5_1/Frameworks/Python.framework/Versio= ns/3.6/lib/python3.6/_strptime.py", line 365, in _strptime > data_string[found.end():]) > ValueError: unconverted data remains: .06 > During handling of the above exception, another exception occurred: > Traceback (most recent call last): > File "scripts/packer-image-manager.py", line 135, in > main() > File "scripts/packer-image-manager.py", line 126, in main > deprecate_images() > File "scripts/packer-image-manager.py", line 56, in deprecate_images > dry_run=3Dargs.dry_run) > File "/Users/fatmcgav/Work/infra/infra/packer2.0/lib/utils.py", line 82, = in deprecate_image > if image.deprecate(replacement_image, 'DEPRECATED', deleted=3Ddelete_at): > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/compute/drivers/gce.py", line 699, in deprecate > deprecated, obsolete, deleted) > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/compute/drivers/gce.py", line 6375, in ex_deprecate_image > attribute) > ValueError: deleted must be an RFC3339 timestamp > {noformat} > It would appear that the date parsing logic in `timestamp_to_datetime` [2= ]=C2=A0is flawed, assuming that the length of `microseconds + tz offset` wi= ll always be 10 characters, whereas the above example has 12 characters. > Patch applied as follows: > {code:java} > diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/g= ce.py > index 3dfc87d3..efcdcb87 100644 > --- a/libcloud/compute/drivers/gce.py > +++ b/libcloud/compute/drivers/gce.py > @@ -52,9 +52,9 @@ def timestamp_to_datetime(timestamp): > :return: Datetime object corresponding to timestamp > :rtype: :class:`datetime.datetime` > """ > - # We remove timezone offset and microseconds (Python 2.5 strptime doesn= 't > - # support %f) > - ts =3D datetime.datetime.strptime(timestamp[:-10], '%Y-%m-%dT%H:%M:%S') > + # We remove timezone offset > + ts =3D datetime.datetime.strptime(timestamp[:timestamp.index('+')], > + '%Y-%m-%dT%H:%M:%S.%f') > tz_hours =3D int(timestamp[-5:-3]) > tz_mins =3D int(timestamp[-2:]) * int(timestamp[-6:-5] + '1') > tz_delta =3D datetime.timedelta(hours=3Dtz_hours, minutes=3Dtz_mins) > {code} > =C2=A0 > However once that issue was resolved, the Google Compute API call failed = with: > {noformat} > 2018-08-20 12:04:59 - DEBUG: Delete at =3D 2018-08-23T11:04:59.525758+00:= 00 > Traceback (most recent call last): > File "scripts/packer-image-manager.py", line 135, in > main() > File "scripts/packer-image-manager.py", line 126, in main > deprecate_images() > File "scripts/packer-image-manager.py", line 56, in deprecate_images > dry_run=3Dargs.dry_run) > File "/Users/fatmcgav/Work/infra/infra/packer2.0/lib/utils.py", line 82, = in deprecate_image > if image.deprecate(replacement_image, 'DEPRECATED', deleted=3Ddelete_at): > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/compute/drivers/gce.py", line 699, in deprecate > deprecated, obsolete, deleted) > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/compute/drivers/gce.py", line 6381, in ex_deprecate_image > self.connection.request(request, method=3D'POST', data=3Dimage_data).obje= ct > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/compute/drivers/gce.py", line 124, in request > response =3D super(GCEConnection, self).request(*args, **kwargs) > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/common/google.py", line 808, in request > *args, **kwargs) > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/common/base.py", line 637, in request > response =3D responseCls(**kwargs) > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/common/base.py", line 155, in __init__ > self.object =3D self.parse_body() > File "/Users/fatmcgav/.virtualenvs/packer_builder/lib/python3.6/site-pack= ages/libcloud/common/google.py", line 288, in parse_body > raise InvalidRequestError(message, self.status, code) > libcloud.common.google.InvalidRequestError: {'domain': 'global', 'reason'= : 'invalid', 'message': "Invalid value for field 'deprecationStatus.deleted= ': '2018-08-23T11:04:59.525758'. Must be a string representation of a full-= date or date-time in valid RFC3339 format with either 0 or 3 digits for fra= ctional seconds"} > {noformat} > Tracked this down to the `ex_deprecate_image`=C2=A0 [3] call, which appea= red to be 'validating' the format of the timestamp, but not actually storin= g the converted result. > Another patch applied: > {code:java} > diff --git a/libcloud/compute/drivers/gce.py b/libcloud/compute/drivers/g= ce.py > index 3dfc87d3..a3329133 100644 > --- a/libcloud/compute/drivers/gce.py > +++ b/libcloud/compute/drivers/gce.py > @@ -6523,11 +6523,11 @@ class GCENodeDriver(NodeDriver): > continue > try: > - timestamp_to_datetime(value) > + value =3D timestamp_to_datetime(value) > except: > raise ValueError('%s must be an RFC3339 timestamp' % > attribute) > - image_data[attribute] =3D value > + image_data[attribute] =3D value.isoformat(timespec=3D'milliseconds') > request =3D '/global/images/%s/deprecate' % (image.name) > {code} > Pull request incoming with both patches. > [1] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drive= rs/gce.py#L6485-L6486] > [2] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drive= rs/gce.py#L44-L61]=C2=A0 > [3] [https://github.com/apache/libcloud/blob/trunk/libcloud/compute/drive= rs/gce.py#L6525-L6530]=C2=A0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)