libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From to...@apache.org
Subject [1/3] libcloud git commit: Ensure RawResponse downloads don't consume RAM
Date Sun, 07 Jan 2018 18:37:32 GMT
Repository: libcloud
Updated Branches:
  refs/heads/trunk f70264647 -> 4732d83be


Ensure RawResponse downloads don't consume RAM

The fix is general but the cause is very specific: since libcloud 2.0,
`download_object()` in the S3 and GCS drivers first places the item in
RAM, then writes it to a file. The reason it broke is that accessing
`body` in a requests Response consumes it entirely and loads it in RAM.
To fix this, I moved `body` to a property: it is still accessibly, but
loaded only if requested.

Closes #1132

Signed-off-by: Tomaz Muraus <tomaz@tomaz.me>


Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo
Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/31411884
Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/31411884
Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/31411884

Branch: refs/heads/trunk
Commit: 31411884cd2f989421e378f0bc8fc0c7b6c4783f
Parents: f702646
Author: Quentin Pradet <quentin@clustree.com>
Authored: Mon Oct 16 14:58:45 2017 +0400
Committer: Tomaz Muraus <tomaz@tomaz.me>
Committed: Sun Jan 7 18:16:32 2018 +0100

----------------------------------------------------------------------
 libcloud/common/base.py | 6 +++++-
 libcloud/http.py        | 4 ++++
 2 files changed, 9 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/libcloud/blob/31411884/libcloud/common/base.py
----------------------------------------------------------------------
diff --git a/libcloud/common/base.py b/libcloud/common/base.py
index 24fdbdb..6dc0589 100644
--- a/libcloud/common/base.py
+++ b/libcloud/common/base.py
@@ -274,12 +274,15 @@ class RawResponse(Response):
         if not self._response:
             response = self.connection.connection.getresponse()
             self._response = HttpLibResponseProxy(response)
-            self.body = response.content
             if not self.success():
                 self.parse_error()
         return self._response
 
     @property
+    def body(self):
+        return self.response.body
+
+    @property
     def reason(self):
         if not self._reason:
             self._reason = self.response.reason
@@ -581,6 +584,7 @@ class Connection(object):
                     url=url,
                     body=data,
                     headers=headers,
+                    raw=raw,
                     stream=stream)
             else:
                 if retry_enabled:

http://git-wip-us.apache.org/repos/asf/libcloud/blob/31411884/libcloud/http.py
----------------------------------------------------------------------
diff --git a/libcloud/http.py b/libcloud/http.py
index 9149995..5e85026 100644
--- a/libcloud/http.py
+++ b/libcloud/http.py
@@ -319,3 +319,7 @@ class HttpLibResponseProxy(object):
     def version(self):
         # requests doesn't expose this
         return '11'
+
+    @property
+    def body(self):
+        return self._response.content


Mime
View raw message