hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13736) Change PathMetadata to hold S3AFileStatus instead of FileStatus.
Date Thu, 20 Oct 2016 19:08:59 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lei (Eddy) Xu updated HADOOP-13736:
-----------------------------------
    Attachment: HADOOP-13736.wip-01.patch

Upload a working in progress patch to make {{PathMetadata}}, {{DirListingMetadata}} and {{MetadataStore}}
as template, and only implements them using {{S3AFileStatus}} in {{s3guard}}.  So it leaves
the room to share the metadata store code with other projects, while it still enforces that
the metadata store implementation in s3guard can directly write in such way:

{code}
public class DynamoDBMetadataStore implements MetadataStore<S3AFileStatus> {
....
{code}

[~liuml07] and [~fabbri] could you please take a look whether it makes sense to your implementations
respectively?  I attached a {{DummyMS}} and its contract tests in this patch to show the basic
usage. 

[~stevel@apache.org] Thanks for the inputs. HDFS-6984 would be nice to have here, so that
we can just store protobuf bytes in dynamo. However the difficulty still exists when it comes
to deserialize these bytes, it needs to know the concret FileStatus class.


> Change PathMetadata to hold S3AFileStatus instead of FileStatus.
> ----------------------------------------------------------------
>
>                 Key: HADOOP-13736
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13736
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>         Attachments: HADOOP-13736.wip-01.patch
>
>
> {{S3AFileStatus}} is implemented differently with {{FileStatus}}, for instance {{S3AFileStatus#isEmptyDirectory()}}
is not implemented in {{FileStatus()}}. And {{access_time}}, {{block_replication}}, {{owner}},
{{group}} and a few other fields are not meaningful in {{S3AFileStatus}}.  
> So in the scope of {{S3guard}}, it should use {{S3AFileStatus}} in  instead of {{FileStatus}}
in {{PathMetadaa}} to avoid casting the types back and forth in S3A. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message