hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andres Perez (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12057) swiftfs rename on partitioned file attempts to consolidate partitions
Date Fri, 29 Apr 2016 00:03:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263293#comment-15263293
] 

Andres Perez commented on HADOOP-12057:
---------------------------------------

I used a Bluemix Object Store, where you don't have access to the swift.conf file to change
the default maximum object size limit. I think this is good because it separate the dependency
between needing to change something in swift for Hadoop to work. This just uses the [Large
Object Direct API|http://docs.openstack.org/developer/swift/overview_large_objects.html#additional-notes]
of Swift, which in theory will allow to store 1TB files in several segments without merging
them, but still providing a single file to download and combine all the pieces together.

The only issues if that is you hdfs dfs -ls swift://container.store/ you will see the file
displaying with size 0, which is also expected from the documentation linked above.   

> swiftfs rename on partitioned file attempts to consolidate partitions
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-12057
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12057
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/swift
>            Reporter: David Dobbins
>            Assignee: David Dobbins
>         Attachments: HADOOP-12057-006.patch, HADOOP-12057-008.patch, HADOOP-12057.007.patch,
HADOOP-12057.patch, HADOOP-12057.patch, HADOOP-12057.patch, HADOOP-12057.patch, HADOOP-12057.patch
>
>
> In the swift filesystem for openstack, a rename operation on a partitioned file uses
the swift COPY operation, which attempts to consolidate all of the partitions into a single
object.  This causes the rename to fail when the total size of all the partitions exceeds
the maximum object size for swift.  Since partitioned files are primarily created to allow
a file to exceed the maximum object size, this bug makes writing to swift extremely unreliable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message