hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitry Mezhensky (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8545) Filesystem Implementation for OpenStack Swift
Date Thu, 07 Feb 2013 07:33:14 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13573274#comment-13573274

Dmitry Mezhensky commented on HADOOP-8545:

|1. Can you instantiate a swift fs with Path#getFileSystem(conf)? This is how a fs should
work since it means that existing code will work just by changing the scheme or defaultFS
to swift.

It instantiates correctly if required configuration present 

| 2. You probably don't want to store the full uri since when a filesystem is obtained for
a Path, it's going to be a full URI of the Path. Should probably do this: this.uri = URI.create(uri.getScheme()+"://"+uri.getAuthority());

I need full url for different containers and swift installations support, for example different
containers: swift://data.rackspace/var/runs and swift://result.rackspace/home/hadoop (data
and result are different containers in Swift)

| 3. I don't understand the contortions to set the initial working directory. Maybe this.workingDir
= getHomeDirectory() like DFS since I think it's doing what you want in a cleaner way.

I'll try to simplify it

| 4. The custom isDirectory behavior is probably better implemented in a FileStatus subclass.
Otherwise, stat = fs.isDirectory(path) != stat.isDirectory()}}

Directory support in Swift is quite complicated, I think we can't do so.

| The main reason I looked at this is integration with security. You seem to have your own
token implementation, but it's not tied into getDelegationToken. If you don't implement that,
I don't think jobs are going to be able to use this fs.

Yes, Swift has it's ouw token implementation.
> Filesystem Implementation for OpenStack Swift
> ---------------------------------------------
>                 Key: HADOOP-8545
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8545
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs
>    Affects Versions: 2.0.3-alpha, 1.1.2
>            Reporter: Tim Miller
>            Assignee: Dmitry Mezhensky
>         Attachments: HADOOP-8545-1.patch, HADOOP-8545-2.patch, HADOOP-8545-3.patch, HADOOP-8545-4.patch,
HADOOP-8545-5.patch, HADOOP-8545-6.patch, HADOOP-8545-7.patch, HADOOP-8545-8.patch, HADOOP-8545-javaclouds-2.patch,
HADOOP-8545.patch, HADOOP-8545.patch
> Add a filesystem implementation for OpenStack Swift object store, similar to the one
which exists today for S3.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message