libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From to...@apache.org
Subject [1/4] libcloud git commit: storage: Fix hash computation performance on upload
Date Mon, 08 Jan 2018 11:11:25 GMT
Repository: libcloud
Updated Branches:
  refs/heads/trunk ad689715e -> 484e02a63


storage: Fix hash computation performance on upload

The storage base driver computes the hash of uploaded files: individual
drivers use it to ensure the reported hash is correct. Before libcloud
2.x, this was done efficiently: the file was only read once, and we were
using `hash.update()` to avoid keeping the whole file in memory. With
the switch to the requests module, both of these optimizations were
removed inadvertently. It turns out the important one is using
`hash.update()`: computing the hash on the whole file in memory is
orders of magnitude slower.

Closes #1135

Signed-off-by: Tomaz Muraus <tomaz@tomaz.me>


Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo
Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/1e955af3
Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/1e955af3
Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/1e955af3

Branch: refs/heads/trunk
Commit: 1e955af39bb799d32808faa7ebe37455b8ad7e9e
Parents: ad68971
Author: Quentin Pradet <quentin@clustree.com>
Authored: Wed Oct 18 11:15:22 2017 +0400
Committer: Tomaz Muraus <tomaz@tomaz.me>
Committed: Mon Jan 8 11:19:41 2018 +0100

----------------------------------------------------------------------
 libcloud/storage/base.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/libcloud/blob/1e955af3/libcloud/storage/base.py
----------------------------------------------------------------------
diff --git a/libcloud/storage/base.py b/libcloud/storage/base.py
index b3de63d..617cfed 100644
--- a/libcloud/storage/base.py
+++ b/libcloud/storage/base.py
@@ -643,9 +643,9 @@ class StorageDriver(BaseDriver):
     def _hash_buffered_stream(self, stream, hasher, blocksize=65536):
         total_len = 0
         if hasattr(stream, '__next__'):
-            data = libcloud.utils.files.exhaust_iterator(iterator=stream)
-            hasher.update(b(data))
-            total_len = len(data)
+            for chunk in libcloud.utils.files.read_in_chunks(iterator=stream):
+                hasher.update(b(chunk))
+                total_len += len(chunk)
             return (hasher.hexdigest(), total_len)
         if not hasattr(stream, '__exit__'):
             for s in stream:


Mime
View raw message