pulsar-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [pulsar] rivernate opened a new issue #8439: Include MD5 hash for server side validation when offloading ledgers
Date Tue, 03 Nov 2020 23:06:54 GMT

rivernate opened a new issue #8439:
URL: https://github.com/apache/pulsar/issues/8439


   **Is your enhancement request related to a problem? Please describe.**
   It doesn't appear that we currently include the `Content-MD5` when we upload the ledgers
using JCloud. While we haven't seen any corruption from offloading so far, it would be nice
to send the `Content-MD5` so that services that support validation could check the integrity
of the upload
   
   **Describe the solution you'd like**
   Looking at the code here: 
   
   https://github.com/apache/pulsar/blob/ec2de258ce39b38311e92bab6882c198070d8975/tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreManagedLedgerOffloader.java#L176
   
   It should be straight forward to add the `Content-MD5` to the ContentMetaData. I have checked
that JCould supports this here:
   https://github.com/apache/jclouds/blob/94f09325ba1442ed64fc0a0eba7fe8b336f621ce/core/src/main/java/org/jclouds/io/MutableContentMetadata.java#L35
   
   And AWS supports the validation of the individual parts of the Multipart upload, see:
   https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html
   
   The biggest downside I can see is that we would need to buffer the InputStream in the broker
to calculate the MD5Hash
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message