incubator-sling-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Sling > Chunked File Upload Support
Date Mon, 18 Feb 2013 13:30:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/2042/9/1/_/styles/combined.css?spaceKey=SLING&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support">Chunked
File Upload Support</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~shgupta">Shashank
Gupta</a>
    </h4>
        <br/>
                         <h4>Changes (6)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-changed-lines" ><span class="diff-changed-words">{toc:outline=true|separator=pipe|maxLevel=<span
class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">2</span><span
class="diff-added-chars"style="background-color: #dfd;">3</span>}</span> <br></td></tr>
            <tr><td class="diff-unchanged" > <br>h1. Chunked File Upload
<br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >Content-Type: <br>Date: &lt;date&gt;
<br></td></tr>
            <tr><td class="diff-changed-lines" >Content-Length: <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">5242880</span>
<span class="diff-added-words"style="background-color: #dfd;">$requestlen</span>
<br></td></tr>
            <tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" > <br>h3. Upload chunk using
PATCH <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">Sling
support three patch formats to upload chunks on sling.  <br># append/binary-patch: Use
this content type if request contain only binary chunk data. If request has additional parameters
that need to be passed, then use application/json-patch or multipart/form-data-patch content
type.  <br>\[request\] <br>{code:title= Upload the first/intermediate chunk request}
<br>PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res
HTTP/1.1 <br>Content-Type: application/octet-stream-patch <br>If-Match: &quot;e0023aa4e&quot;
<br>Content-Length: $requestlen <br>$binarydata <br>{code} <br></td></tr>
            <tr><td class="diff-unchanged" > <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">\[response\]
<br>{code:title= First/Intermediate chunk upload response} <br>HTTP/1.1 201 CREATED
<br>Location: /content/dam/dam-folder/catalog.pdf.chunk <br>ETag: &quot;e0023aa4f&quot;
<br>Connection: close <br>Server: localhost:4502 <br>{code} <br> <br>#
application/json-patch: JSON format patch document to pass upload attributes along with binary
chunk   <br>\[request\] <br>{code:title= Upload the first/intermediate chunk request}
<br>PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res
HTTP/1.1 <br>Content-Type: application/json-patch <br>If-Match: &quot;e0023aa4e&quot;
<br>Content-Length: $requestlen <br>[ <br>  { <br>	&quot;jcr:data&quot;:
&quot;$binarydata&quot;, <br>	&quot;fileName&quot;: &quot;catalog.pdf&quot;,
<br>	&quot; field1&quot;: &quot;$field1&quot; <br>  } <br>]
<br> <br>{code} <br> <br>\[response\] <br>{code:title= First/Intermediate
chunk upload response} <br>HTTP/1.1 201 CREATED <br>Location: /content/dam/dam-folder/catalog.pdf.chunk
<br>ETag: &quot;e0023aa4f&quot; <br>Connection: close <br>Server:
localhost:4502 <br> <br>{code} <br> <br># multipart/form-data-patch:
multipart/form-data-patch format to pass upload attributes along with binary chunk   <br>\[request\]
<br>{code:title= Upload the first/intermediate chunk request} <br>PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res
HTTP/1.1 <br>Content-Type: multipart/form-data-patch <br>If-Match: &quot;e0023aa4e&quot;
<br>Content-Length: $requestlen <br> <br>--AaB03x <br>content-disposition:
form-data; name=&quot;field1&quot; <br> <br>$field1 <br>--AaB03x
<br>content-disposition: form-data; name=&quot;field2&quot; <br> <br>$field2
<br>--AaB03x <br>content-disposition: form-data; name=&quot;userfile&quot;;
filename=&quot;$filename&quot; <br>Content-Type: $mimetype <br>Content-Transfer-Encoding:
binary <br> <br>$binarydata <br>--AaB03x-- <br> <br>{code} <br>
<br>\[response\] <br>{code:title= First/Intermediate chunk upload response} <br>HTTP/1.1
201 CREATED <br>Location: /content/dam/dam-folder/catalog.pdf.chunk <br>ETag:
&quot;e0023aa4f&quot; <br>Connection: close <br>Server: localhost:4502
<br> <br> <br></td></tr>
            <tr><td class="diff-unchanged" >h3. Query upload  <br> <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" > <br>\[response\] <br></td></tr>
            <tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">{code:title=
Get erroneous last successful upload chunk response} <br></td></tr>
            <tr><td class="diff-added-lines" style="background-color: #dfd;">{code:title=
Chunked upload not found response} <br></td></tr>
            <tr><td class="diff-unchanged" >HTTP/1.1 404/Not Found <br>{code}
<br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <div>
<ul>
    <li><span class='TOCOutline'>1</span> <a href='#ChunkedFileUploadSupport-ChunkedFileUpload'>Chunked
File Upload</a></li>
    <li><span class='TOCOutline'>2</span> <a href='#ChunkedFileUploadSupport-UseCases%3A'>Use
Cases:</a></li>
    <li><span class='TOCOutline'>3</span> <a href='#ChunkedFileUploadSupport-Approach'>Approach</a></li>
    <li><span class='TOCOutline'>4</span> <a href='#ChunkedFileUploadSupport-ProtocolSpecification'>Protocol
Specification</a></li>
<ul>
    <li><span class='TOCOutline'>4.1</span> <a href='#ChunkedFileUploadSupport-Advertisingchunkuploadsupportinsling'>Advertising
chunk upload support in sling</a></li>
    <li><span class='TOCOutline'>4.2</span> <a href='#ChunkedFileUploadSupport-UploadchunkusingPOST'>Upload
chunk using POST</a></li>
    <li><span class='TOCOutline'>4.3</span> <a href='#ChunkedFileUploadSupport-UploadchunkusingPATCH'>Upload
chunk using PATCH</a></li>
    <li><span class='TOCOutline'>4.4</span> <a href='#ChunkedFileUploadSupport-Abortchunkedupload'>Abort
chunked upload</a></li>
</ul>
</ul></div>

<h1><a name="ChunkedFileUploadSupport-ChunkedFileUpload"></a>Chunked File
Upload</h1>

<p>Status: DRAFT<br/>
Created: 20. January 2013<br/>
Author: shgupta<br/>
JIRA: <a href="https://issues.apache.org/jira/browse/SLING-2707" class="external-link"
rel="nofollow">SLING-2707</a><br/>
References: - <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html"
class="external-link" rel="nofollow">http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html</a><br/>
Updated: -</p>

<h1><a name="ChunkedFileUploadSupport-UseCases%3A"></a>Use Cases:</h1>
<p>1. Large file upload - With high speed internet connections, advent of cloud and
HD going mainstream, Sling support large files (&gt; 2GB) upload.<br/>
2. Fault tolerant uploads - Sling provides capability to resume upload from failure point.
It doesn't require client to restart the complete upload process. </p>

<h1><a name="ChunkedFileUploadSupport-Approach"></a>Approach</h1>
<p>Sling provides an endpoint which accepts file chunks in accordance with a specified
protocol. Sling client slices the file in chunks, and upload the chunks in serial manner to
server. Each chunk is numbered serially and the last chunk has additional identifier to distinguish
from the rest chunks. Sling endpoint upon receiving the last chunk, stitches all chunks into
a single file. The file inputstream is passed to inner layers for further processing.</p>

<p>In case of upload failures, sling provides support to query the last chunk uploaded
till failure point. Client resumes chunk upload from last failure point. </p>

<h1><a name="ChunkedFileUploadSupport-ProtocolSpecification"></a>Protocol
Specification </h1>

<h3><a name="ChunkedFileUploadSupport-Advertisingchunkuploadsupportinsling"></a>Advertising
chunk upload support in sling</h3>
<p>Sling advertise its support for chunk upload by adding POST and PATCH methods to
the listing of allowed methods in the "Allow" OPTIONS response header defined in HTTP/1.1.<br/>
[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>OPTION request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
OPTIONS /content/dam/dam-folder/catalog.pdf.chunk HTTP/1.1
</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>OPTION response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200 OK
Allow: GET, PUT, POST, OPTIONS, HEAD, DELETE, PATCH
Accept-Patch: append/binary-patch, application/json-patch, application/json-patch
</pre>
</div></div>
<p><a href="http://greenbytes.de/tech/webdav/rfc5789.html#accept-patch" class="external-link"
rel="nofollow">Accept-Patch</a> used to specify the patch document formats accepted
by the server. Chunk upload supports three  append/binary-patch, application/json-patch, application/json-patch
formats.</p>

<h3><a name="ChunkedFileUploadSupport-UploadchunkusingPOST"></a>Upload chunk
using POST </h3>

<h4><a name="ChunkedFileUploadSupport-Uploadoffirstandintermediatechunks"></a>Upload
of first and intermediate chunks</h4>
<p>Client uses POST method to upload binary chunk of file. <br/>
[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload the first/intermediate chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
POST /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res HTTP/1.1
Host: localhost:4502
If-Match: <span class="code-quote">"e0023aa4e"</span>
Content-Type:
Date: &lt;date&gt;
Content-Length: $requestlen
</pre>
</div></div>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First/Intermediate chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201 CREATED
Content-Length: 0
ETag: <span class="code-quote">"e0023aa4f"</span>
Location: /content/dam/dam-folder/catalog.pdf.chunk
Connection: close
Server: localhost:4502
</pre>
</div></div>
<p>The location header provides path to retrieve chunk upload attributes.<br/>
Concurrent request from multiple client requests  will corrupt the resource. Clients should
use a conditional request such that the request will fail if the resource has been updated
since the client last accessed the resource. For example, the client can use a strong <a
href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.19" class="external-link"
rel="nofollow">ETag</a> in an <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.24"
class="external-link" rel="nofollow">If-Match</a>  header on the chunk request.</p>

<h4><a name="ChunkedFileUploadSupport-Uploadoflastchunk"></a>Upload of last
chunk</h4>
<p>Client appends "/last" to upload url to signify that it would be last that needs
to be uploaded. The last chunk upload request can optionally be multipart request if client
requires to send additional parameters to sling.</p>

<p>[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload last chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
POST /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res/last HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
Content-type: multipart/form-data, boundary=AaB03x

--AaB03x
content-disposition: form-data; name=<span class="code-quote">"fileSize"</span>

210000000
--AaB03x
content-disposition: form-data; name=<span class="code-quote">"file"</span>; fileName=<span
class="code-quote">"catalog.pdf"</span>
Content-Type: application/pdf
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--

</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload last chunk response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201/CREATED
Content-Length: 0
Location: /content/dam/dam-folder/catalog.pdf
Content-MD5: $md5_hash
Connection: close
</pre>
</div></div>
<p>The location header provides location of fully uploaded asset.The response contains
the $md5_hash of merged binary. </p>

<h3><a name="ChunkedFileUploadSupport-UploadchunkusingPATCH"></a>Upload
chunk using PATCH</h3>
<p>Sling support three patch formats to upload chunks on sling. </p>
<ol>
	<li>append/binary-patch: Use this content type if request contain only binary chunk
data. If request has additional parameters that need to be passed, then use application/json-patch
or multipart/form-data-patch content type.<br/>
[request]
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload the first/intermediate chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res HTTP/1.1
Content-Type: application/octet-stream-patch
If-Match: <span class="code-quote">"e0023aa4e"</span>
Content-Length: $requestlen
$binarydata
</pre>
</div></div></li>
</ol>


<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First/Intermediate chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: <span class="code-quote">"e0023aa4f"</span>
Connection: close
Server: localhost:4502
</pre>
</div></div>

<ol>
	<li>application/json-patch: JSON format patch document to pass upload attributes along
with binary chunk<br/>
[request]
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload the first/intermediate chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res HTTP/1.1
Content-Type: application/json-patch
If-Match: <span class="code-quote">"e0023aa4e"</span>
Content-Length: $requestlen
[
  {
	<span class="code-quote">"jcr:data"</span>: <span class="code-quote">"$binarydata"</span>,
	<span class="code-quote">"fileName"</span>: <span class="code-quote">"catalog.pdf"</span>,
	<span class="code-quote">" field1"</span>: <span class="code-quote">"$field1"</span>
  }
]

</pre>
</div></div></li>
</ol>


<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First/Intermediate chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: <span class="code-quote">"e0023aa4f"</span>
Connection: close
Server: localhost:4502

</pre>
</div></div>

<ol>
	<li>multipart/form-data-patch: multipart/form-data-patch format to pass upload attributes
along with binary chunk<br/>
[request]
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload the first/intermediate chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res HTTP/1.1
Content-Type: multipart/form-data-patch
If-Match: <span class="code-quote">"e0023aa4e"</span>
Content-Length: $requestlen

--AaB03x
content-disposition: form-data; name=<span class="code-quote">"field1"</span>

$field1
--AaB03x
content-disposition: form-data; name=<span class="code-quote">"field2"</span>

$field2
--AaB03x
content-disposition: form-data; name=<span class="code-quote">"userfile"</span>;
filename=<span class="code-quote">"$filename"</span>
Content-Type: $mimetype
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--

</pre>
</div></div></li>
</ol>


<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First/Intermediate chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: <span class="code-quote">"e0023aa4f"</span>
Connection: close
Server: localhost:4502


h3. Query upload 

h4. Query to retrieve last successful chunk upload
Returns the number of last chunk upload and bytes uploaded successfully on sling. 
\[request\]
{code:title= Get last successful chunk upload request}
GET /content/dam/dam-folder/catalog.pdf.chunk.json HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
Content-Length: 0
</pre>
</div></div>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Get last successful upload chunk response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200/OK
Connection: close

{
    <span class="code-quote">"chunkNumber"</span>: 8
    <span class="code-quote">"bytesuploaded"</span>: 19872
}
</pre>
</div></div>

<p>If no broken/discontinued upload sling returns 404 not found response. </p>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Chunked upload not found response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 404/Not Found
</pre>
</div></div>

<h3><a name="ChunkedFileUploadSupport-Abortchunkedupload"></a>Abort chunked
upload</h3>
<p>[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Abort chunked upload Request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
DELETE /content/dam/dam-folder/catalog.pdf.chunk.res HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
</pre>
</div></div>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Abort chunked upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200/OK
Connection: close
</pre>
</div></div>


    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;">
            <a href="https://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>
        </div>
        <a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=30749779&revisedVersion=17&originalVersion=16">View
Changes</a>
                |
        <a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message