hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Virajith Jalaparti (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-10637) Modifications to remove the assumption that FsVolumes are backed by java.io.File.
Date Sat, 16 Jul 2016 00:31:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380362#comment-15380362
] 

Virajith Jalaparti edited comment on HDFS-10637 at 7/16/16 12:30 AM:
---------------------------------------------------------------------

The patch contains the following changes (text is incorporated from comments in HDFS-9809):

# Instead of associating an FsVolume with a base path (which is a {{java.io.File}}), we associate
it with a {{StorageLocation}}. This allows removing the dependence on {{java.io.File}} and
replacing it with a more general one, which can point to a {{java.io.File}} or an abstract
{{URI}} representing an external storage. Using {{StorageLocation}} instead of defining a
new type for location allows us to reuse its functionality and plug into the rest of the code
easily. Following this intuition, we replaced {{FsVolumeSpi#getBasePath}} with {{FsVolumeSpi#getStorageLocation}}.
As a result, comparisons and references to FsVolumes which were done using the {{java.io.File}}
returned by {{FsVolumeSpi#getBasePath}} are now replaced by comparisons and references to
the {{StorageLocation}} returned by {{FsVolumeSpi#getStorageLocation}}. 
# Similarly, the patch also replaces calls to get*Dir on FsVolumeImpl. In particular, the
{{DirectoryScanner.ReportCompiler}} calls on {{FsVolumeSpi#getFinalizedDir}} and compiles
the report assuming that this returns a {{java.io.File}}. However, in HDFS-9806, data may
not be stored in files. Further, the {{DirectoryScanner.ReportCompiler#compileReport}} function
assumes the way blocks are stored in FsVolumes which can be different for different {{FsVolumeSpi}}
implementations. To address these assumptions and to hide the details on how volumes implement
block storage, {{ReportCompiler#compileReport}} is moved to {{FsVolumeSpi}}.
# A {{FsVolumeImplBuilder}} is added and calls to the constructor of {{FsVolumeImpl}} are
replaced with those to the builder. The idea behind this is to construct the appropriate volume
based on the {{StorageLocation}} (specified using {{dfs.datanode.data.dir}}). For example,
as part of HDFS-9806, if a {{StorageLocation}} is of a PROVIDED type, we would construct a
{{ProvidedVolumeImpl}}. Otherwise, a {{FsVolumeImpl}} would be built. 



was (Author: virajith):
The patch contains the following changes (some text is incorporated from comments in HDFS-9809):

# Instead of associating an FsVolume with a base path (which is a {{java.io.File}}), we associate
it with a {{StorageLocation}}. This allows removing the dependence on {{java.io.File}} and
replacing it with a more general one, which can point to a {{java.io.File}} or an abstract
{{URI}} representing an external storage. Using {{StorageLocation}} instead of defining a
new type for location allows us to reuse its functionality and plug into the rest of the code
easily. Following this intuition, we replaced {{FsVolumeSpi#getBasePath}} with {{FsVolumeSpi#getStorageLocation}}.
As a result, comparisons and references to FsVolumes which were done using the {{java.io.File}}
returned by {{FsVolumeSpi#getBasePath}} are now replaced by comparisons and references to
the {{StorageLocation}} returned by {{FsVolumeSpi#getStorageLocation}}. 
# Similarly, the patch also replaces calls to get*Dir on FsVolumeImpl. In particular, the
{{DirectoryScanner.ReportCompiler}} calls on {{FsVolumeSpi#getFinalizedDir}} and compiles
the report assuming that this returns a {{java.io.File}}. However, in HDFS-9806, data may
not be stored in files. Further, the {{DirectoryScanner.ReportCompiler#compileReport}} function
assumes the way blocks are stored in FsVolumes which can be different for different {{FsVolumeSpi}}
implementations. To address these assumptions and to hide the details on how volumes implement
block storage, {{ReportCompiler#compileReport}} is moved to {{FsVolumeSpi}}.
# A {{FsVolumeImplBuilder}} is added and calls to the constructor of {{FsVolumeImpl}} are
replaced with those to the builder. The idea behind this is to construct the appropriate volume
based on the {{StorageLocation}} (specified using {{dfs.datanode.data.dir}}). For example,
as part of HDFS-9806, if a {{StorageLocation}} is of a PROVIDED type, we would construct a
{{ProvidedVolumeImpl}}. Otherwise, a {{FsVolumeImpl}} would be built. 


> Modifications to remove the assumption that FsVolumes are backed by java.io.File.
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-10637
>                 URL: https://issues.apache.org/jira/browse/HDFS-10637
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, fs
>            Reporter: Virajith Jalaparti
>         Attachments: HDFS-10637.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message