hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11968) ViewFS: StoragePolicies commands fail with HDFS federation
Date Sat, 23 Sep 2017 09:44:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16177689#comment-16177689
] 

Mukul Kumar Singh commented on HDFS-11968:
------------------------------------------

Hi [~arpitagarwal], I ran the following commands on a viewfs,hdfs and webhdfs based instances
and following are the list of commands and their output.

ViewFS based uri
{code}
*core-site.xml*
 <property>
    <name>fs.default.name</name>
    <value>viewfs://cluster1</value>
    </description>
    <final>true</final>
  </property>
  <property>
    <name>fs.viewfs.mounttable.cluster1.link./home</name>
    <value>hdfs://127.0.0.1:8020/dir1</value>
  </property>
  <property>
    <name>fs.viewfs.mounttable.cluster1.link./tmp</name>
    <value>hdfs://127.0.0.1:8020/dir2</value>
  </property>

*list storage policies*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -listPolicies
Block Storage Policies:
	BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
	BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE],
replicationFallbacks=[DISK, ARCHIVE]}
	BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
	BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD,
DISK]}
	BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
	BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK],
replicationFallbacks=[DISK]}

*set storage policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -setStoragePolicy
-path /home -policy ALL_SSD
Set storage policy ALL_SSD on /home

*get storage policies after setting a policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -getStoragePolicy
-path /home
The storage policy of /home:
BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]

*unset storage policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -unsetStoragePolicy
-path /home
Unset storage policy from /home
{code}

Web HDFS based uri
{code}
*core site*
    <name>fs.default.name</name>
    <value>webhdfs://127.0.0.1:9870/</value>

*list storage policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -listPolicies
Block Storage Policies:
	BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
	BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE],
replicationFallbacks=[DISK, ARCHIVE]}
	BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
	BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD,
DISK]}
	BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
	BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK],
replicationFallbacks=[DISK]}

*set storage policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -setStoragePolicy
-path /dir2 -policy WARM
Set storage policy WARM on /dir2

*get storage policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -getStoragePolicy
-path /dir2
The storage policy of /dir2:
BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE],
replicationFallbacks=[DISK, ARCHIVE]}
{code}

HDFS based uri
{code}
*core site*
    <name>fs.default.name</name>
    <value>hdfs://127.0.0.1:8020/</value>

*list storage policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -listPolicies
Block Storage Policies:
	BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
	BlockStoragePolicy{WARM:5, storageTypes=[DISK, ARCHIVE], creationFallbacks=[DISK, ARCHIVE],
replicationFallbacks=[DISK, ARCHIVE]}
	BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
	BlockStoragePolicy{ONE_SSD:10, storageTypes=[SSD, DISK], creationFallbacks=[SSD, DISK], replicationFallbacks=[SSD,
DISK]}
	BlockStoragePolicy{ALL_SSD:12, storageTypes=[SSD], creationFallbacks=[DISK], replicationFallbacks=[DISK]}
	BlockStoragePolicy{LAZY_PERSIST:15, storageTypes=[RAM_DISK, DISK], creationFallbacks=[DISK],
replicationFallbacks=[DISK]}

*set storage policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -setStoragePolicy
-path /dir1 -policy COLD
Set storage policy COLD on /dir1

*get storage policy*
[root@304a057b260f opt]# /opt/hadoop-3.1.0-SNAPSHOT/bin/hdfs storagepolicies -getStoragePolicy
-path /dir1
The storage policy of /dir1:
BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[], replicationFallbacks=[]}
{code}

> ViewFS: StoragePolicies commands fail with HDFS federation
> ----------------------------------------------------------
>
>                 Key: HDFS-11968
>                 URL: https://issues.apache.org/jira/browse/HDFS-11968
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs
>    Affects Versions: 2.7.1
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>         Attachments: HDFS-11968.001.patch, HDFS-11968.002.patch, HDFS-11968.003.patch,
HDFS-11968.004.patch, HDFS-11968.005.patch, HDFS-11968.006.patch, HDFS-11968.007.patch, HDFS-11968.008.patch,
HDFS-11968.009.patch, HDFS-11968.010.patch
>
>
> hdfs storagepolicies command fails with HDFS federation.
> For storage policies commands, a given user path should be resolved to a HDFS path and
> storage policy command should be applied onto the resolved HDFS path.
> {code}
>   static DistributedFileSystem getDFS(Configuration conf)
>       throws IOException {
>     FileSystem fs = FileSystem.get(conf);
>     if (!(fs instanceof DistributedFileSystem)) {
>       throw new IllegalArgumentException("FileSystem " + fs.getUri() +
>           " is not an HDFS file system");
>     }
>     return (DistributedFileSystem)fs;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message