Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Wed, 1 Jul 2015 11:42:05 +0000 (UTC)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12830314.1431724714000.69796.1435750925273@Atlassian.JIRA>
In-Reply-To: <JIRA.12830314.1431724714000@Atlassian.JIRA>
References: <JIRA.12830314.1431724714000@Atlassian.JIRA>
 <JIRA.12830314.1431724714717@arcas>
Subject: [jira] [Commented] (HADOOP-11981) Add storage policy APIs to
 filesystem docs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609975#comment-14609975 ] 

Steve Loughran commented on HADOOP-11981:
-----------------------------------------

It's not yet ready to go in, because it's more javadocs than specification. These documents constitute an attempt to define what a filesystem is expected to do in way from which we can derive tests and implementations. For that they have to be unambiguous and not omit things which were felt to be "too obvious to mention". Its actually quite hard to do; for the files in there I had to go through all the implementations we had and see what they really did ... tagging as "newbie" is actually quite optimistic. 

The storage policy APIs are going to have be another aspect of the filesystem model covered in {{model.md}}. that is currently, the FS is
{code}
    (Directories:set[Path], Files:[Path:List[byte]], Symlinks:set[Path])
{code}

It's going to have to become something like

{code}
    (Directories:set[Path], Files:[Path:List[byte]], Symlinks:set[Path], storagePolicies: set[BlockStoragePolicySpi], storagePolicy[Path: String])
{code}

The Xatrrs can go in at the same time.

Operations on the storage policy then become actions which read/write these new sets & maps, things we can write down in the python-base notation:

{code}
   getStoragePolicies()

preconditions:
   if FS.storagePolicies=={} raise UnsupportedOperationException

postconditions:

   result = storagePolicies(FS)
{code}

A more interesting one is the setter:

{code}
   setStoragePolicy(src, policyName)

preconditions:
   if FS.storagePolicies=={} raise UnsupportedOperationException
   if [p in FS.storagePolicies where p.getName==policyName] ==[] raise HadoopIllegalArgumentException

postconditions:

   FS' = FS where FS'.storagePolicy(src)==policyName
{code}

What this does is try to make things less ambiguous, and so make implementation of filesystems and the tests easier. It also means that we can look at the HDFS implementation and say whether or not this is what it should be doing.

 For Example
# {{BlockStoragePolicySuite}} is actually using case insensitive checks without specifying the locale ... that's exactly the kind of thing we need to be defining, and so use it to identify issues like HDFS-8705.
# {{FSDirAttrOp}} will raise an {{IOE}} if the storage policy is disabled. This is something which clearly needs specifying
# HDFS implies the caller needs write access to the path. Again, this need to be part of the specification.


Accordingly: not yet, and it's going to be harder than you expect. I will help review it for you, and help define that extended FS model that's needed.


> Add storage policy APIs to filesystem docs
> ------------------------------------------
>
>                 Key: HADOOP-11981
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11981
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>              Labels: newbie
>         Attachments: HADOOP-11981.incomplete.01.patch
>
>
> HDFS-8345 exposed the storage policy APIs via the FileSystem.
> The FileSystem docs should be updated accordingly.
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)