hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11981) Add storage policy APIs to filesystem docs
Date Wed, 01 Jul 2015 11:42:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609975#comment-14609975
] 

Steve Loughran commented on HADOOP-11981:
-----------------------------------------

It's not yet ready to go in, because it's more javadocs than specification. These documents
constitute an attempt to define what a filesystem is expected to do in way from which we can
derive tests and implementations. For that they have to be unambiguous and not omit things
which were felt to be "too obvious to mention". Its actually quite hard to do; for the files
in there I had to go through all the implementations we had and see what they really did ...
tagging as "newbie" is actually quite optimistic. 

The storage policy APIs are going to have be another aspect of the filesystem model covered
in {{model.md}}. that is currently, the FS is
{code}
    (Directories:set[Path], Files:[Path:List[byte]], Symlinks:set[Path])
{code}

It's going to have to become something like

{code}
    (Directories:set[Path], Files:[Path:List[byte]], Symlinks:set[Path], storagePolicies:
set[BlockStoragePolicySpi], storagePolicy[Path: String])
{code}

The Xatrrs can go in at the same time.

Operations on the storage policy then become actions which read/write these new sets &
maps, things we can write down in the python-base notation:

{code}
   getStoragePolicies()

preconditions:
   if FS.storagePolicies=={} raise UnsupportedOperationException

postconditions:

   result = storagePolicies(FS)
{code}

A more interesting one is the setter:

{code}
   setStoragePolicy(src, policyName)

preconditions:
   if FS.storagePolicies=={} raise UnsupportedOperationException
   if [p in FS.storagePolicies where p.getName==policyName] ==[] raise HadoopIllegalArgumentException

postconditions:

   FS' = FS where FS'.storagePolicy(src)==policyName
{code}

What this does is try to make things less ambiguous, and so make implementation of filesystems
and the tests easier. It also means that we can look at the HDFS implementation and say whether
or not this is what it should be doing.

 For Example
# {{BlockStoragePolicySuite}} is actually using case insensitive checks without specifying
the locale ... that's exactly the kind of thing we need to be defining, and so use it to identify
issues like HDFS-8705.
# {{FSDirAttrOp}} will raise an {{IOE}} if the storage policy is disabled. This is something
which clearly needs specifying
# HDFS implies the caller needs write access to the path. Again, this need to be part of the
specification.


Accordingly: not yet, and it's going to be harder than you expect. I will help review it for
you, and help define that extended FS model that's needed.


> Add storage policy APIs to filesystem docs
> ------------------------------------------
>
>                 Key: HADOOP-11981
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11981
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>              Labels: newbie
>         Attachments: HADOOP-11981.incomplete.01.patch
>
>
> HDFS-8345 exposed the storage policy APIs via the FileSystem.
> The FileSystem docs should be updated accordingly.
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message