libcloud-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [libcloud] jfharden opened a new issue #1401: Uploading to s3 encrypted with a custom KMS key fails
Date Mon, 06 Jan 2020 12:01:41 GMT
jfharden opened a new issue #1401: Uploading to s3 encrypted with a custom KMS key fails
URL: https://github.com/apache/libcloud/issues/1401
 
 
   ## Summary
   
   When uploading files to s3 where the bucket has server side encryption with a custom KMS
key the upload fails
   
   ## Detailed Information
   Python 3.7
   Libcloud: 2.8.0
   OS: Amazon Linux 2
   
   Libcloud is inspecting the etag expecting it to be the md5sum of the uploaded data object
but this is not always the case (where using a custom KMS key, or doing a multi-part upload):
https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html
   
   > Objects created by the PUT Object, POST Object, or Copy operation, or through the
AWS Management Console, and are encrypted by SSE-C or SSE-KMS, have ETags that are not an
MD5 digest of their object data.
   >
   > If an object is created by either the Multipart Upload or Part Copy operation, the
ETag is not an MD5 digest, regardless of the method of encryption.
   
   the s3 storage driver is comparing the etag to the hash calculated locally on the streamed
file (https://github.com/apache/libcloud/blob/trunk/libcloud/storage/drivers/s3.py#L850) but
where the etag is not an md5 hash of the file this will always fail
   
   I've included a stack trace below from cassandra-medusa (https://github.com/thelastpickle/cassandra-medusa)
which led to me investigating this problem
   
   ```
   Traceback (most recent call last):
     File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 274, in main
       cassandra, node_backup, storage, differential_mode, config)
     File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 320, in do_backup
       num_files = backup_snapshots(storage, manifest, node_backup, node_backup_cache, snapshot)
     File "/usr/local/lib/python3.7/site-packages/medusa/backup.py", line 388, in backup_snapshots
       manifest_objects = storage.storage_driver.upload_blobs(needs_backup, dst_path)
     File "/usr/local/lib/python3.7/site-packages/medusa/storage/s3_storage.py", line 95,
in upload_blobs
       multi_part_upload_threshold=int(self.config.multi_part_upload_threshold),
     File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py",
line 87, in upload_blobs
       return job.execute(list(src))
     File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py",
line 51, in execute
       return list(executor.map(self.with_storage, iterables))
     File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
       yield fs.pop().result()
     File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 435, in result
       return self.__get_result()
     File "/usr/lib64/python3.7/concurrent/futures/_base.py", line 384, in __get_result
       raise self._exception
     File "/usr/lib64/python3.7/concurrent/futures/thread.py", line 57, in run
       result = self.fn(*self.args, **self.kwargs)
     File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py",
line 60, in with_storage
       return self.func(self.storage, connection, iterable)
     File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py",
line 83, in <lambda>
       storage, connection, src_file, dest, bucket, multi_part_upload_threshold
     File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py",
line 119, in __upload_file
       obj = _upload_single_part(connection, src, bucket, full_object_name)
     File "/usr/local/lib/python3.7/site-packages/retrying.py", line 49, in wrapped_f
       return Retrying(*dargs, **dkw).call(f, *args, **kw)
     File "/usr/local/lib/python3.7/site-packages/retrying.py", line 212, in call
       raise attempt.get()
     File "/usr/local/lib/python3.7/site-packages/retrying.py", line 247, in get
       six.reraise(self.value[0], self.value[1], self.value[2])
     File "/usr/local/lib/python3.7/site-packages/six.py", line 696, in reraise
       raise value
     File "/usr/local/lib/python3.7/site-packages/retrying.py", line 200, in call
       attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
     File "/usr/local/lib/python3.7/site-packages/medusa/storage/aws_s3_storage/concurrent.py",
line 127, in _upload_single_part
       os.fspath(src), container=bucket, object_name=object_name
     File "/usr/local/lib/python3.7/site-packages/libcloud/storage/drivers/s3.py", line 492,
in upload_object
       storage_class=ex_storage_class)
     File "/usr/local/lib/python3.7/site-packages/libcloud/storage/drivers/s3.py", line 854,
in _put_object
       object_name=object_name, driver=self)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message