hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewan Higgs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6708) StorageType should be encoded in the block token
Date Wed, 25 Jan 2017 15:20:26 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15837893#comment-15837893
] 

Ewan Higgs commented on HDFS-6708:
----------------------------------

Hi all. I took this work over from Pieter. I've got some questions about how we want this
to work.

First, if I understand correctly we want to check the token mainly in the DataXceiver, passing
the token and the locally available StorageTypes whenever we call {{DataXceiver.checkAccess}}
(which ends up calling {{BlockTokenSecretManager.checkAccess}}) for writes only (i.e. {{replaceBlock}},
{{transferBlock}}, {{writeBlock}} - and *not* {{blockChecksum}}, {{blockGroupChecksum}}, or
{{requestShortCircuitFds}}). 

If we're only checking the storage type when we write data then we use the passed in {{StorageType}}.
However, if we want to do this on reads as well, we need to gather the available {{StorageType[]}}.
We can get the locally available storage types as follows:

{code}
  // only needed if we want to use StorageType for reads as well...
  private static StorageType[] getStorageTypes(DataNode datanode) {
    final FsDatasetSpi dataset =  datanode.getFSDataset();
    final FsDatasetSpi.FsVolumeReferences vols;
    vols = dataset.getFsVolumeReferences();

    List<StorageType> storageTypes = new ArrayList<StorageType>(vols.size());
    Iterator<FsVolumeSpi> iter = vols.iterator();
    while (iter.hasNext()) {
      FsVolumeSpi vol = iter.next();
      StorageType storageType = vol.getStorageType();
      if (storageType != null) {
        storageTypes.add(storageType);
      }
    }
    return storageTypes.toArray(new StorageType[0]);
  }
{code}

The resulting check uses the Token {{StorageType[]}} and compares it to the {{StorageType[]}}
passed in by the Protobuf request operation. I think the rules should be as follows:

||{{Token StorageType[]}}||{{Node StorageType[]}}|| Result ||
|\* | null| Error|
|null | \* | Error|
| {{\[\]}} | {{\[\]}} | Not OK (maybe Error?) |
| {{\[DISK\]}} | {{\[DISK\]}} | OK |
| {{\[DISK\]}} | {{\[\]}} | Not OK | 
| {{\[SSD, DISK\]}} | {{\[DISK\]}} | Not OK|
| {{\[SSD, DISK\]}} | {{\[SSD, DISK\]}} | OK |
| {{\[\]}} | {{\[SSD, DISK\]}} | OK |

Finally, I found that {{TestBalancer#testBalancerWithKeytabs}} and {{TestMover#testMoverWithKeytabs}}
fail because they create a cluster with {{\[DISK, ARCHIVE\]}} storage; set the StoragePolicy
to {{COLD}} (which has no {{DISK}}) and then try to run the Balancer or Mover. This fails
since the token has {{\[DISK\]}} as the {{StorageType}} and the request has {{\[ARCHIVE\]}}.
Perhaps the token is stale. 

> StorageType should be encoded in the block token
> ------------------------------------------------
>
>                 Key: HDFS-6708
>                 URL: https://issues.apache.org/jira/browse/HDFS-6708
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode, namenode
>    Affects Versions: 2.4.1
>            Reporter: Arpit Agarwal
>            Assignee: Pieter Reuse
>
> HDFS-6702 is adding support for file creation based on StorageType.
> The block token is used as a tamper-proof channel for communicating block parameters
from the NN to the DN during block creation. The StorageType should be included in this block
token.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message