From notifications-return-14166-archive-asf-public=cust-asf.ponee.io@libcloud.apache.org Sun Jan 7 19:37:37 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 38B99180654 for ; Sun, 7 Jan 2018 19:37:37 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 28B22160C2A; Sun, 7 Jan 2018 18:37:37 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6F161160C14 for ; Sun, 7 Jan 2018 19:37:36 +0100 (CET) Received: (qmail 35413 invoked by uid 500); 7 Jan 2018 18:37:35 -0000 Mailing-List: contact notifications-help@libcloud.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@libcloud.apache.org Delivered-To: mailing list notifications@libcloud.apache.org Received: (qmail 35404 invoked by uid 500); 7 Jan 2018 18:37:35 -0000 Delivered-To: apmail-libcloud-commits@libcloud.apache.org Received: (qmail 35401 invoked by uid 99); 7 Jan 2018 18:37:35 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 07 Jan 2018 18:37:35 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 518CEDFFCF; Sun, 7 Jan 2018 18:37:32 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: tomaz@apache.org To: commits@libcloud.apache.org Date: Sun, 07 Jan 2018 18:37:32 -0000 Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: [1/3] libcloud git commit: Ensure RawResponse downloads don't consume RAM Repository: libcloud Updated Branches: refs/heads/trunk f70264647 -> 4732d83be Ensure RawResponse downloads don't consume RAM The fix is general but the cause is very specific: since libcloud 2.0, `download_object()` in the S3 and GCS drivers first places the item in RAM, then writes it to a file. The reason it broke is that accessing `body` in a requests Response consumes it entirely and loads it in RAM. To fix this, I moved `body` to a property: it is still accessibly, but loaded only if requested. Closes #1132 Signed-off-by: Tomaz Muraus Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/31411884 Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/31411884 Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/31411884 Branch: refs/heads/trunk Commit: 31411884cd2f989421e378f0bc8fc0c7b6c4783f Parents: f702646 Author: Quentin Pradet Authored: Mon Oct 16 14:58:45 2017 +0400 Committer: Tomaz Muraus Committed: Sun Jan 7 18:16:32 2018 +0100 ---------------------------------------------------------------------- libcloud/common/base.py | 6 +++++- libcloud/http.py | 4 ++++ 2 files changed, 9 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/libcloud/blob/31411884/libcloud/common/base.py ---------------------------------------------------------------------- diff --git a/libcloud/common/base.py b/libcloud/common/base.py index 24fdbdb..6dc0589 100644 --- a/libcloud/common/base.py +++ b/libcloud/common/base.py @@ -274,12 +274,15 @@ class RawResponse(Response): if not self._response: response = self.connection.connection.getresponse() self._response = HttpLibResponseProxy(response) - self.body = response.content if not self.success(): self.parse_error() return self._response @property + def body(self): + return self.response.body + + @property def reason(self): if not self._reason: self._reason = self.response.reason @@ -581,6 +584,7 @@ class Connection(object): url=url, body=data, headers=headers, + raw=raw, stream=stream) else: if retry_enabled: http://git-wip-us.apache.org/repos/asf/libcloud/blob/31411884/libcloud/http.py ---------------------------------------------------------------------- diff --git a/libcloud/http.py b/libcloud/http.py index 9149995..5e85026 100644 --- a/libcloud/http.py +++ b/libcloud/http.py @@ -319,3 +319,7 @@ class HttpLibResponseProxy(object): def version(self): # requests doesn't expose this return '11' + + @property + def body(self): + return self._response.content