hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen He (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-12471) Support Swift file (> 5GB) continuious uploading where there is a failure
Date Fri, 09 Oct 2015 06:55:26 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-12471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chen He updated HADOOP-12471:
-----------------------------
    Description: 
Current Swift FileSystem supports file larger than 5GB. 
File will be chunked as large as 4.6GB (configurable). For example, if there is a 46GB file
"foo" in swift, 
Then the structure will look like:

foo/000001
foo/000002
foo/000003
...
foo/000010

User will not see those 00000x files if they don't specify. That means, if user does:
\> hadoop fs -ls swift://container.serviceProvidor/foo

It only shows:
dwr-r--r--    4.6GB    foo

However, in my test, if there is a failure, during uploading the foo file, the previous uploaded
chunks will be left in the object store. It will be good to support continuous uploading based
on previous leftover

  was:
Current Swift FileSystem supports file larger than 5GB. 
File will be chunked as large as 4.6GB (configurable). For example, if there is a 46GB file
"foo" in swift, 
Then the structure will look like:

foo/000001
foo/000002
foo/000003
...
foo/000010

User will not see those 00000x files if they don't specify. That means, if use do:
\> hadoop fs -ls swift://container.serviceProvidor/foo

It only shows:
dwr-r--r--    4.6GB    foo

However, in my test, if there is a failure, during uploading the foo file, the previous uploaded
chunks will be left in the object store. It will be good to support continuous uploading based
on previous leftover


> Support Swift file (> 5GB) continuious uploading where there is a failure
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-12471
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12471
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/swift
>    Affects Versions: 2.7.1
>            Reporter: Chen He
>
> Current Swift FileSystem supports file larger than 5GB. 
> File will be chunked as large as 4.6GB (configurable). For example, if there is a 46GB
file "foo" in swift, 
> Then the structure will look like:
> foo/000001
> foo/000002
> foo/000003
> ...
> foo/000010
> User will not see those 00000x files if they don't specify. That means, if user does:
> \> hadoop fs -ls swift://container.serviceProvidor/foo
> It only shows:
> dwr-r--r--    4.6GB    foo
> However, in my test, if there is a failure, during uploading the foo file, the previous
uploaded chunks will be left in the object store. It will be good to support continuous uploading
based on previous leftover



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message