incubator-sling-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Sling > Chunked File Upload Support
Date Mon, 18 Feb 2013 11:11:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/2042/9/1/_/styles/combined.css?spaceKey=SLING&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support">Chunked
File Upload Support</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~shgupta">Shashank
Gupta</a>
    </h4>
        <br/>
                         <h4>Changes (20)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">{toc:outline=true|separator=pipe|maxLevel=40}</span>
<span class="diff-added-words"style="background-color: #dfd;">{toc:outline=true|separator=pipe|maxLevel=2}</span>
<br></td></tr>
            <tr><td class="diff-unchanged" > <br>h1. Chunked File Upload
<br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >h1. Protocol Specification  <br>
<br></td></tr>
            <tr><td class="diff-changed-lines" ><span class="diff-changed-words">h<span
class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">2</span><span
class="diff-added-chars"style="background-color: #dfd;">3</span>.</span> Advertising
chunk upload support in sling <br></td></tr>
            <tr><td class="diff-unchanged" >Sling advertise its support for chunk
upload by adding POST and PATCH methods to the listing of allowed methods in the &quot;Allow&quot;
OPTIONS response header defined in HTTP/1.1. <br>\[request\] <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >h4. Upload of first and intermediate
chunks <br>Client uses POST method to upload binary chunk of file.  <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[request]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[request\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= Upload the first/intermediate
chunk request} <br>POST /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res
HTTP/1.1 <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[response]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[response\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= First/Intermediate chunk
upload response} <br>HTTP/1.1 201 CREATED <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >Client appends &quot;/last&quot;
to upload url to signify that it would be last that needs to be uploaded. The last chunk upload
request can optionally be multipart request if client requires to send additional parameters
to sling. <br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[request]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[request\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= Upload last chunk request}
<br>POST /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res/last
HTTP/1.1 <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" > <br>{code} <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[response]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[response\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= Upload last chunk response}
<br>HTTP/1.1 201/CREATED <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >h4. Query to retrieve last successful
chunk upload <br>Returns the number of last chunk upload and bytes uploaded successfully
on sling.  <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[request]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[request\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= Get last successful chunk
upload request} <br>GET /content/dam/dam-folder/catalog.pdf.chunk.json HTTP/1.1 <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[response]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[response\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= Get last successful upload
chunk response} <br>HTTP/1.1 200/OK <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >If no broken/discontinued upload sling
returns 404 not found response.  <br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[response]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[response\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= Get erroneous last successful
upload chunk response} <br>HTTP/1.1 404/Not Found <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" > <br>h3. Abort chunked upload
<br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[request]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[request\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= Abort chunked upload
Request} <br>DELETE /content/dam/dam-folder/catalog.pdf.chunk.res HTTP/1.1 <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">[response]
<br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[response\]
<br></td></tr>
            <tr><td class="diff-unchanged" >{code:title= Abort chunked upload
response} <br>HTTP/1.1 200/OK <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <div>
<ul>
    <li><span class='TOCOutline'>1</span> <a href='#ChunkedFileUploadSupport-ChunkedFileUpload'>Chunked
File Upload</a></li>
    <li><span class='TOCOutline'>2</span> <a href='#ChunkedFileUploadSupport-UseCases%3A'>Use
Cases:</a></li>
    <li><span class='TOCOutline'>3</span> <a href='#ChunkedFileUploadSupport-Approach'>Approach</a></li>
    <li><span class='TOCOutline'>4</span> <a href='#ChunkedFileUploadSupport-ProtocolSpecification'>Protocol
Specification</a></li>
</ul></div>

<h1><a name="ChunkedFileUploadSupport-ChunkedFileUpload"></a>Chunked File
Upload</h1>

<p>Status: DRAFT<br/>
Created: 20. January 2013<br/>
Author: shgupta<br/>
JIRA: <a href="https://issues.apache.org/jira/browse/SLING-2707" class="external-link"
rel="nofollow">SLING-2707</a><br/>
References: - <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html"
class="external-link" rel="nofollow">http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html</a><br/>
Updated: -</p>

<h1><a name="ChunkedFileUploadSupport-UseCases%3A"></a>Use Cases:</h1>
<p>1. Large file upload - With high speed internet connections, advent of cloud and
HD going mainstream, Sling support large files (&gt; 2GB) upload.<br/>
2. Fault tolerant uploads - Sling provides capability to resume upload from failure point.
It doesn't require client to restart the complete upload process. </p>

<h1><a name="ChunkedFileUploadSupport-Approach"></a>Approach</h1>
<p>Sling provides an endpoint which accepts file chunks in accordance with a specified
protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to
server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish
from the rest chunks. Sling endpoint upon receiving the last chunk, stitches all chunks into
a single file. The file inputstream is passed to inner layers for further processing.</p>

<p>In case of upload failures, sling provides support to query the last chunk uploaded
till failure point. Client resumes chunk upload from last failure point. </p>

<h1><a name="ChunkedFileUploadSupport-ProtocolSpecification"></a>Protocol
Specification </h1>

<h3><a name="ChunkedFileUploadSupport-Advertisingchunkuploadsupportinsling"></a>Advertising
chunk upload support in sling</h3>
<p>Sling advertise its support for chunk upload by adding POST and PATCH methods to
the listing of allowed methods in the "Allow" OPTIONS response header defined in HTTP/1.1.<br/>
[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>OPTION request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
OPTIONS /content/dam/dam-folder/catalog.pdf.chunk HTTP/1.1
</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>OPTION response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200 OK
Allow: GET, PUT, POST, OPTIONS, HEAD, DELETE, PATCH
Accept-Patch: append/binary-patch, application/json-patch, application/json-patch
</pre>
</div></div>
<p><a href="http://greenbytes.de/tech/webdav/rfc5789.html#accept-patch" class="external-link"
rel="nofollow">Accept-Patch</a> used to specify the patch document formats accepted
by the server. Chunk upload supports three  append/binary-patch, application/json-patch, application/json-patch
formats.</p>

<h3><a name="ChunkedFileUploadSupport-UploadchunkusingPOST"></a>Upload chunk
using POST </h3>

<h4><a name="ChunkedFileUploadSupport-Uploadoffirstandintermediatechunks"></a>Upload
of first and intermediate chunks</h4>
<p>Client uses POST method to upload binary chunk of file. <br/>
[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload the first/intermediate chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
POST /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res HTTP/1.1
Host: localhost:4502
If-Match: <span class="code-quote">"e0023aa4e"</span>
Content-Type:
Date: &lt;date&gt;
Content-Length: 5242880
</pre>
</div></div>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First/Intermediate chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201 CREATED
Content-Length: 0
ETag: <span class="code-quote">"e0023aa4f"</span>
Location: /content/dam/dam-folder/catalog.pdf.chunk
Connection: close
Server: localhost:4502
</pre>
</div></div>
<p>The location header provides path to retrieve chunk upload attributes.<br/>
Concurrent request from multiple client requests  will corrupt the resource. Clients should
use a conditional request such that the request will fail if the resource has been updated
since the client last accessed the resource. For example, the client can use a strong <a
href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.19" class="external-link"
rel="nofollow">ETag</a> in an <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.24"
class="external-link" rel="nofollow">If-Match</a>  header on the chunk request.</p>

<h4><a name="ChunkedFileUploadSupport-Uploadoflastchunk"></a>Upload of last
chunk</h4>
<p>Client appends "/last" to upload url to signify that it would be last that needs
to be uploaded. The last chunk upload request can optionally be multipart request if client
requires to send additional parameters to sling.</p>

<p>[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload last chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
POST /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res/last HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
Content-type: multipart/form-data, boundary=AaB03x

--AaB03x
content-disposition: form-data; name=<span class="code-quote">"fileSize"</span>

210000000
--AaB03x
content-disposition: form-data; name=<span class="code-quote">"file"</span>; fileName=<span
class="code-quote">"catalog.pdf"</span>
Content-Type: application/pdf
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--

</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload last chunk response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201/CREATED
Content-Length: 0
Location: /content/dam/dam-folder/catalog.pdf
Content-MD5: $md5_hash
Connection: close
</pre>
</div></div>
<p>The location header provides location of fully uploaded asset.The response contains
the $md5_hash of merged binary. </p>

<h3><a name="ChunkedFileUploadSupport-UploadchunkusingPATCH"></a>Upload
chunk using PATCH</h3>

<h3><a name="ChunkedFileUploadSupport-Queryupload"></a>Query upload </h3>

<h4><a name="ChunkedFileUploadSupport-Querytoretrievelastsuccessfulchunkupload"></a>Query
to retrieve last successful chunk upload</h4>
<p>Returns the number of last chunk upload and bytes uploaded successfully on sling.
<br/>
[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Get last successful chunk upload request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
GET /content/dam/dam-folder/catalog.pdf.chunk.json HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
Content-Length: 0
</pre>
</div></div>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Get last successful upload chunk response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200/OK
Connection: close

{
    <span class="code-quote">"chunkNumber"</span>: 8
    <span class="code-quote">"bytesuploaded"</span>: 19872
}
</pre>
</div></div>

<p>If no broken/discontinued upload sling returns 404 not found response. </p>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Get erroneous last successful upload chunk response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 404/Not Found
</pre>
</div></div>

<h3><a name="ChunkedFileUploadSupport-Abortchunkedupload"></a>Abort chunked
upload</h3>
<p>[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Abort chunked upload Request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
DELETE /content/dam/dam-folder/catalog.pdf.chunk.res HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
</pre>
</div></div>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Abort chunked upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200/OK
Connection: close
</pre>
</div></div>


    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;">
            <a href="https://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>
        </div>
        <a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=30749779&revisedVersion=16&originalVersion=15">View
Changes</a>
                |
        <a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message