hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-13448) S3Guard: Define MetadataStore interface.
Date Thu, 01 Sep 2016 21:33:21 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-13448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Aaron Fabbri updated HADOOP-13448:
----------------------------------
    Attachment: HADOOP-13448-HADOOP-13345.002.patch

I'm posting a v2 patch which evolves the initial patch some.  @cnauorth 
is on vacation, but I wanted to keep the process moving with some suggested
modifications I made (for my work on subtask HADOOP-13452):

- Define concrete types PathMetadata and DirListingMetadata.
  - Both are "containers" of FileStatuses.  I use a type bound to allow them 
    to hold any subclass of FileStatus, e.g. S3AFileStatus.
  - DirListingMetadata is a collection of FileStatus plus an
    isAuthoritative flag as recently discussed in HADOOP-13345.
  - I thought about having a common base class but it felt contrived, as things
    like isAuthoratitive(), lookup/get/delete() only make sense for directory
    listings, not single path metadata.  That is, I didn't see a common "is a"
    relationship.
    
- I removed MetadataStore's dependency on S3AFileSystem for the Logger.  I'm
  hoping this code can eventually be factored out of s3a and become useful for
  other FS clients.
  
- Added another put() method to MetadataStore that takes a DirListingMetadata.
  I explain the need for the isAuthorative bit on put() in my recent comment
  on HADOOP-13345
  
Question / Next Steps:

- Should MetadataStore be an interface? I left it an abstract class but
  it looks more like an Interface at the moment.  Perhaps we'll want to add
  back a common Logger object, that doesn't depend on S3AFileSystem? 
  
- How do folks feel about this API consuming/producing FileStatus'?  I
  think it simplifies the client code quite a bit, which is one of my 
  top concerns.  I'm fine with changing it if it becomes awkward.
  
- I'm wondering if DirListingMetadata will become a collection of PathMetadata
  instead of FileStatus'.  For now I did the latter, thinking of what FS clients
  will need: FileStatus'.  If we add things like content hashes or etags in 
  future I could see this becoming a collection of PathMetadata instead.

> S3Guard: Define MetadataStore interface.
> ----------------------------------------
>
>                 Key: HADOOP-13448
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13448
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: HADOOP-13448-HADOOP-13345.001.patch, HADOOP-13448-HADOOP-13345.002.patch
>
>
> Define the common interface for metadata store operations.  This is the interface that
any metadata back-end must implement in order to integrate with S3Guard.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message