openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Marth <mma...@adobe.com.INVALID>
Subject Re: AttachmentStore - Handling concurrent attachment updates
Date Mon, 26 Mar 2018 16:48:58 GMT
Hi Chetan,



My2c: making the attachments immutable will yield great benefits as you write below:



    1. Proper handling of concurrent updates

    2. Simplified caching of attachments as immutable objects can be cached easily



On #2: with immutable attachments caching becomes trivial which will help with more distributed
deployments (across different data centers).



Great proposal

Michael









´╗┐On 26/03/18 05:58, "Chetan Mehrotra" <chetan.mehrotra@gmail.com> wrote:



    Last week I had a Slack call with Rodric around AttachmentStore PR and

    as part of that we discussed the problem around handling of concurrent

    updates of attachments. Details below are based on that discussion.



    As of now CouchDB can detect concurrent updates of attachment due to

    inbuilt MVCC support. However most of the Object Stores (like S3/IBM

    COS, Azure Blob Store etc) which are to be used for new

    AttachmentStore implementation does not provide any conditional update

    and are designed more for immutable storage.



    Consider an Action Update sequence which is currently done in 2 parts



    1. Update the document

    2. Upload the attachment



    Now consider an AttachmentStore implementation (as per PR #3453

    design) which stores attachment content against a key like



       whiskentity/<doc id>/<attachment name>



    Where



    1. whiskaction - Key prefix to store attachments related to Whisk entities

    2. <doc id> - Document Id with which the attachment is being attached

    3. <attachment name> - Name of attachment like `jarfile`



    Object Stores are optimized for direct key lookup and also allows

    searches based on key prefix. Hence the use of such a format which

    allows direct attachment lookup for readAttachment and all attachments

    related to specific doc for deleteAttachments



    Now consider following flow



    1. thread 1: updates the document and succeeds

    2. thread 2: updates the document (based on thread 1) and succeeds

    3. thread 2: attaches i.e. writes an attachment to the AttachmentStore

    4. thread 1: attaches



    This would result in a race condition where in the end attachment

    meant for document state at #1 gets linked to document at state #2. To

    handle such cases we should switch to immutable attachment design



    A - Proposal - Use Immutable Attachments

    ----------------------------------------------------



    In current flow we perform an "update" of existing attachment with a

    given name. For e.g. currently action update flow is like



    1. Put document with attachment info



        "exec": {

            "kind": "java",

            "code": {

              "attachmentName": "jarfile",

              "attachmentType": "application/java-archive"

            },

            "binary": true,

            "main": "Hello"

          }



    2. Attach the attachment with name set to value of `attachmentName`



    Instead of that we should allow `ArtifactStore` (which in turn rely on

    AttachmentStore) to generate the name and then save that name against

    `attachmentName`. So proposed flow is



    1. Upload the attachment and have ArtifactStore return a generated name



      protected[core] def attach(doc: DocInfo, contentType: ContentType,

    docStream: Source[ByteString, _])(

        implicit transid: TransactionId): Future[(DocInfo, AttachmentName)]



    2. Then update the document with attachmentName set to name returned

    in previous step



    3. Then delete the old attachment after #2 completes successfully



    With this approach the attachments would be immutable and that would enable



    1. Proper handling of concurrent updates

    2. Simplified caching of attachments as immutable objects can be cached easily



    B - Orphaned Blob Garbage Collection

    ----------------------------------------------



    With above approach there is a possiblity that some action update flow

    may end up in between leaving some orphan blob instances in Object

    stores. To clean them up we can implement a garbage collection login

    as part of wskadmin



    Please share your feedback about the new proposal. I would start work

    on a PR for new proposal so that its easier to discuss specific

    semantics. Once this work is done we can come back to AttachmentStore

    PR and implement that as per newer flow



    Chetan Mehrotra


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message