incubator-sling-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conflue...@apache.org
Subject [CONF] Apache Sling > Chunked File Upload Support
Date Thu, 28 Feb 2013 14:19:00 GMT
<html>
<head>
    <base href="https://cwiki.apache.org/confluence">
            <link rel="stylesheet" href="/confluence/s/2042/9/1/_/styles/combined.css?spaceKey=SLING&amp;forWysiwyg=true"
type="text/css">
    </head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
    <h2><a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support">Chunked
File Upload Support</a></h2>
    <h4>Page <b>edited</b> by             <a href="https://cwiki.apache.org/confluence/display/~shgupta">Shashank
Gupta</a>
    </h4>
        <br/>
                         <h4>Changes (2)</h4>
                                 
    
<div id="page-diffs">
                    <table class="diff" cellpadding="0" cellspacing="0">
    
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >\[request\] <br>{code:title=
First chunk upload request} <br></td></tr>
            <tr><td class="diff-changed-lines" >POST /content/dam <span class="diff-added-words"style="background-color:
#dfd;"> HTTP/1.1</span> <br></td></tr>
            <tr><td class="diff-unchanged" >--ULPTM_fXARgFjsvsyvaO_W-2loSNQw <br>Content-Disposition:
form-data; name=&quot;:chunkNumber&quot; <br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
            <tr><td class="diff-unchanged" >&lt;/html&gt; <br>{code}
<br></td></tr>
            <tr><td class="diff-changed-lines" ><span class="diff-added-words"style="background-color:
#dfd;">*</span> The location tag contains unique chunk upload which need to be passed
on subsequent request. <span class="diff-added-words"style="background-color: #dfd;">*</span>
<br></td></tr>
            <tr><td class="diff-unchanged" > <br>h4. Upload of last chunk
<br></td></tr>
            <tr><td class="diff-snipped" >...<br></td></tr>
    
            </table>
    </div>                            <h4>Full Content</h4>
                    <div class="notificationGreySide">
        <div>
<ul>
    <li><span class='TOCOutline'>1</span> <a href='#ChunkedFileUploadSupport-ChunkedFileUpload'>Chunked
File Upload</a></li>
    <li><span class='TOCOutline'>2</span> <a href='#ChunkedFileUploadSupport-UseCases%3A'>Use
Cases:</a></li>
    <li><span class='TOCOutline'>3</span> <a href='#ChunkedFileUploadSupport-Approach'>Approach</a></li>
    <li><span class='TOCOutline'>4</span> <a href='#ChunkedFileUploadSupport-ProtocolSpecification'>Protocol
Specification</a></li>
<ul>
    <li><span class='TOCOutline'>4.1</span> <a href='#ChunkedFileUploadSupport-UploadchunkusingPOST'>Upload
chunk using POST</a></li>
    <li><span class='TOCOutline'>4.2</span> <a href='#ChunkedFileUploadSupport-UploadchunkusingPATCH'>Upload
chunk using PATCH</a></li>
    <li><span class='TOCOutline'>4.3</span> <a href='#ChunkedFileUploadSupport-Querytoretrievelastsuccessfulchunkupload'>Query
to retrieve last successful chunk upload</a></li>
    <li><span class='TOCOutline'>4.4</span> <a href='#ChunkedFileUploadSupport-Abortchunkedupload'>Abort
chunked upload</a></li>
</ul>
</ul></div>

<h1><a name="ChunkedFileUploadSupport-ChunkedFileUpload"></a>Chunked File
Upload</h1>

<p>Status: DRAFT<br/>
Created: 20. January 2013<br/>
Author: shgupta<br/>
JIRA: <a href="https://issues.apache.org/jira/browse/SLING-2707" class="external-link"
rel="nofollow">SLING-2707</a><br/>
References: - <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html"
class="external-link" rel="nofollow">http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html</a><br/>
Updated: -</p>

<h1><a name="ChunkedFileUploadSupport-UseCases%3A"></a>Use Cases:</h1>
<p>1. Large file upload - With high speed internet connections, advent of cloud and
HD going mainstream, Sling support large files (&gt; 2GB) upload.<br/>
2. Fault tolerant uploads - Sling provides capability to resume upload from failure point.
It doesn't require client to restart the complete upload process. </p>

<h1><a name="ChunkedFileUploadSupport-Approach"></a>Approach</h1>
<p>Sling provides an extension to <a href="http://sling.apache.org/site/manipulating-content-the-slingpostservlet-servletspost.html"
class="external-link" rel="nofollow">SlingPostServlet</a> which accepts file chunks
in accordance with a specified protocol. Sling client slices the file in chunks, and upload
the chunks in serial manner to server. Each chunk is numbered serially and the last chunk
has additional identifier to distinguish from the rest chunks. Sling <a href="http://sling.apache.org/site/manipulating-content-the-slingpostservlet-servletspost.html"
class="external-link" rel="nofollow">SlingPostServlet</a>upon receiving the last
chunk, stitches all chunks into a single file and creates node tree in repository. </p>

<p>In case of upload failures, sling provides support to query the last chunk uploaded
till failure point. Client resumes chunk upload from last failure point. </p>

<h1><a name="ChunkedFileUploadSupport-ProtocolSpecification"></a>Protocol
Specification </h1>


<h3><a name="ChunkedFileUploadSupport-UploadchunkusingPOST"></a>Upload chunk
using POST </h3>

<h4><a name="ChunkedFileUploadSupport-Firstchunkupload"></a>First chunk
upload</h4>
<p>Client uses POST method to parent path to upload binary chunk of file. Client passes
":chunkNumber" and "fileName" as multipart request parameters.<br/>
[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First chunk upload request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
POST /content/dam  HTTP/1.1
--ULPTM_fXARgFjsvsyvaO_W-2loSNQw
Content-Disposition: form-data; name=<span class="code-quote">":chunkNumber"</span>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

1
--ULPTM_fXARgFjsvsyvaO_W-2loSNQw
Content-Disposition: form-data; name=<span class="code-quote">"*"</span>; filename=<span
class="code-quote">"$fileName"</span>
Content-Type: application/octet-stream
Content-Transfer-Encoding: binary
$binarydata
</pre>
</div></div>

<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200 OK
&lt;html&gt;
&lt;head&gt;
    &lt;title&gt;Content modified /tmp/sling/chunks&lt;/title&gt;
&lt;/head&gt;
    &lt;body&gt;
    &lt;h1&gt;Content modified /tmp/sling/chunks&lt;/h1&gt;
    &lt;table&gt;
        &lt;tbody&gt;
            &lt;tr&gt;
                &lt;td&gt;Status&lt;/td&gt;
                &lt;td&gt;&lt;div id=<span class="code-quote">"Status"</span>&gt;200&lt;/div&gt;&lt;/td&gt;
            &lt;/tr&gt;
            &lt;tr&gt;
                &lt;td&gt;Message&lt;/td&gt;
                &lt;td&gt;&lt;div id=<span class="code-quote">"Message"</span>&gt;OK&lt;/div&gt;&lt;/td&gt;
            &lt;/tr&gt;
            &lt;tr&gt;
                &lt;td&gt;Location&lt;/td&gt;
                &lt;td&gt;&lt;a href=<span class="code-quote">"/tmp/sling/chunks/e622fd2ebaa14b69b5a5b51a32084a6c"</span>
id=<span class="code-quote">"Location"</span>&gt;/tmp/sling/chunks/e622fd2ebaa14b69b5a5b51a32084a6c&lt;/a&gt;&lt;/td&gt;
            &lt;/tr&gt;
            &lt;tr&gt;
                &lt;td&gt;Parent Location&lt;/td&gt;
                &lt;td&gt;&lt;a href=<span class="code-quote">"/tmp/sling/chunks"</span>
id=<span class="code-quote">"ParentLocation"</span>&gt;/tmp/sling/chunks&lt;/a&gt;&lt;/td&gt;
            &lt;/tr&gt;
            &lt;tr&gt;
                &lt;td&gt;Path&lt;/td&gt;
                &lt;td&gt;&lt;div id=<span class="code-quote">"Path"</span>&gt;/tmp/sling/chunks&lt;/div&gt;&lt;/td&gt;
            &lt;/tr&gt;
            &lt;tr&gt;
                &lt;td&gt;Referer&lt;/td&gt;
                &lt;td&gt;&lt;a href=<span class="code-quote">"" id="</span>Referer"&gt;&lt;/a&gt;&lt;/td&gt;
            &lt;/tr&gt;
            &lt;tr&gt;
                &lt;td&gt;ChangeLog&lt;/td&gt;
                &lt;td&gt;&lt;div id=<span class="code-quote">"ChangeLog"</span>&gt;&amp;lt;pre&amp;gt;&amp;lt;/pre&amp;gt;&lt;/div&gt;&lt;/td&gt;
            &lt;/tr&gt;
        &lt;/tbody&gt;
    &lt;/table&gt;
    &lt;p&gt;&lt;a href=""&gt;Go Back&lt;/a&gt;&lt;/p&gt;
    &lt;p&gt;&lt;a href=<span class="code-quote">"/tmp/sling/chunks/e622fd2ebaa14b69b5a5b51a32084a6c"</span>&gt;Modified
Resource&lt;/a&gt;&lt;/p&gt;
    &lt;p&gt;&lt;a href=<span class="code-quote">"/tmp/sling/chunks"</span>&gt;Parent
of Modified Resource&lt;/a&gt;&lt;/p&gt;
    &lt;/body&gt;
&lt;/html&gt;
</pre>
</div></div>
<ul>
	<li>The location tag contains unique chunk upload which need to be passed on subsequent
request. *</li>
</ul>


<h4><a name="ChunkedFileUploadSupport-Uploadoflastchunk"></a>Upload of last
chunk</h4>
<p>Client appends "/last" to upload url to signify that it would be last that needs
to be uploaded. The last chunk upload request can optionally be multipart request if client
requires to send additional parameters to sling.</p>

<p>[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload last chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
POST /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res/last HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
Content-type: multipart/form-data, boundary=AaB03x

--AaB03x
content-disposition: form-data; name=<span class="code-quote">"fileSize"</span>

210000000
--AaB03x
content-disposition: form-data; name=<span class="code-quote">"file"</span>; fileName=<span
class="code-quote">"catalog.pdf"</span>
Content-Type: application/pdf
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--

</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload last chunk response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201/CREATED
Content-Length: 0
Location: /content/dam/dam-folder/catalog.pdf
Content-MD5: $md5_hash
Connection: close
</pre>
</div></div>
<p>The location header provides location of fully uploaded asset.The response contains
the $md5_hash of merged binary. </p>

<h3><a name="ChunkedFileUploadSupport-UploadchunkusingPATCH"></a>Upload
chunk using PATCH</h3>
<p>Sling support three patch formats to upload chunks on sling. All other semantics
remain same as  <a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support#ChunkedFileUploadSupport-UploadchunkusingPOST"
class="external-link" rel="nofollow">Upload chunk using POST</a></p>
<ol>
	<li>append/binary-patch: Use this content type if request contain only binary chunk
data. If request has additional parameters that need to be passed, then use application/json-patch
or multipart/form-data-patch content type.<br/>
[request]
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload the first/intermediate chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res HTTP/1.1
Content-Type: application/octet-stream-patch
If-Match: <span class="code-quote">"e0023aa4e"</span>
Content-Length: $requestlen
$binarydata
</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First/Intermediate chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: <span class="code-quote">"e0023aa4f"</span>
Connection: close
Server: localhost:4502
</pre>
</div></div></li>
	<li>application/json-patch: JSON format patch document to pass upload attributes along
with binary chunk<br/>
[request]
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload the first/intermediate chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res HTTP/1.1
Content-Type: application/json-patch
If-Match: <span class="code-quote">"e0023aa4e"</span>
Content-Length: $requestlen
[
  {
	<span class="code-quote">"jcr:data"</span>: <span class="code-quote">"$binarydata"</span>,
	<span class="code-quote">"fileName"</span>: <span class="code-quote">"catalog.pdf"</span>,
	<span class="code-quote">" field1"</span>: <span class="code-quote">"$field1"</span>
  }
]
</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First/Intermediate chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: <span class="code-quote">"e0023aa4f"</span>
Connection: close
Server: localhost:4502
</pre>
</div></div></li>
	<li>multipart/form-data-patch: multipart/form-data-patch format to pass upload attributes
along with binary chunk<br/>
[request]
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Upload the first/intermediate chunk request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
PATCH /content/dam/dam-folder/catalog.pdf.chunk.&lt;chunk_number&gt;.res HTTP/1.1
Content-Type: multipart/form-data-patch
If-Match: <span class="code-quote">"e0023aa4e"</span>
Content-Length: $requestlen

--AaB03x
content-disposition: form-data; name=<span class="code-quote">"field1"</span>

$field1
--AaB03x
content-disposition: form-data; name=<span class="code-quote">"field2"</span>

$field2
--AaB03x
content-disposition: form-data; name=<span class="code-quote">"userfile"</span>;
filename=<span class="code-quote">"$filename"</span>
Content-Type: $mimetype
Content-Transfer-Encoding: binary

$binarydata
--AaB03x--
</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>First/Intermediate chunk upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 201 CREATED
Location: /content/dam/dam-folder/catalog.pdf.chunk
ETag: <span class="code-quote">"e0023aa4f"</span>
Connection: close
Server: localhost:4502
</pre>
</div></div></li>
</ol>


<h3><a name="ChunkedFileUploadSupport-Querytoretrievelastsuccessfulchunkupload"></a>Query
to retrieve last successful chunk upload</h3>
<p>Returns the number of last chunk upload and bytes uploaded successfully on sling.
<br/>
[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Get last successful chunk upload request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
GET /content/dam/dam-folder/catalog.pdf.chunk.json HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
Content-Length: 0
</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Get last successful upload chunk response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200/OK
Connection: close

{
    <span class="code-quote">"chunkNumber"</span>: 8
    <span class="code-quote">"bytesuploaded"</span>: 19872
}
</pre>
</div></div>
<p>If no broken/discontinued upload sling returns 404 not found response. <br/>
[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Chunked upload not found response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 404/Not Found
</pre>
</div></div>

<h3><a name="ChunkedFileUploadSupport-Abortchunkedupload"></a>Abort chunked
upload</h3>
<p>[request]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Abort chunked upload Request</b></div><div
class="codeContent panelContent">
<pre class="code-java">
DELETE /content/dam/dam-folder/catalog.pdf.chunk.res HTTP/1.1
Host: localhost:4502
Date: &lt;date&gt;
</pre>
</div></div>
<p>[response]</p>
<div class="code panel" style="border-width: 1px;"><div class="codeHeader panelHeader"
style="border-bottom-width: 1px;"><b>Abort chunked upload response</b></div><div
class="codeContent panelContent">
<pre class="code-java">
HTTP/1.1 200/OK
Connection: close
</pre>
</div></div>


    </div>
        <div id="commentsSection" class="wiki-content pageSection">
        <div style="float: right;">
            <a href="https://cwiki.apache.org/confluence/users/viewnotifications.action"
class="grey">Change Notification Preferences</a>
        </div>
        <a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support">View
Online</a>
        |
        <a href="https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=30749779&revisedVersion=23&originalVersion=22">View
Changes</a>
                |
        <a href="https://cwiki.apache.org/confluence/display/SLING/Chunked+File+Upload+Support?showComments=true&amp;showCommentArea=true#addcomment">Add
Comment</a>
            </div>
</div>
</div>
</div>
</div>
</body>
</html>

Mime
View raw message