hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewan Higgs (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-11639) [READ] Encode the BlockAlias in the client protocol
Date Wed, 17 May 2017 12:28:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-11639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ewan Higgs updated HDFS-11639:
------------------------------
    Attachment: HDFS-11639-HDFS-9806.003.patch

Attaching an updated patch that addresses some of [~virajith]'s comments:

{quote}
    
    - In ProvidedBlocksBuilder#newLocatedBlock, the fileRegion should be
      resolved only if the block has PROVIDED locations (i.e., hasProvidedLocations
      is true). When dfs.namenode.provided.enabled is set to true, all LocatedBlock
      are created in this method, and for non-provided blocks, a resolution of
      BlockAlias is needed.
    - PBHelperClient#convertLocatedBlockProto() and
      PBHelperClient#convertLocatedBlock() should be modified to decode/encode the
      BlockAlias bytes.
    - How about decoding the blockAlias bytes in DataXceiver#readBlock using
      a new DataTransferProtoUtil#blockAliasFromProto(bytes[] blockAlias method
      instead of using the BlockAlias#builder()? The former will be in-line with
      the way the protobufs are decoded in DataXceiver. Further, if in the
      future a different BlockAlias is used, the current implementation of using
      the FileRegion#Builder in DataXceiver#readBlock will be hard to extend
      (will end up being try FileRegion#Builder, if null try
      BlockAliasXX#Builder and so on).
    - Similar to passing on BlockAlias from DataXceiver#readBlock to
      BlockSender, it should be passed along from DataXceiver#readBlock to
      BlockReceiver. However, we would not need it till we have writes
      implemented.
    - DataStreamer#blockAlias will never be non-null. I think it should
      be initialized in DFSOutputStream.
{quote}

This also added BlockAlias to the transferBlocks message but it isn't incorporated in since
this is a write message. e.g. {{DNA_TRANSFER}} has not yet been updated.

> [READ] Encode the BlockAlias in the client protocol
> ---------------------------------------------------
>
>                 Key: HDFS-11639
>                 URL: https://issues.apache.org/jira/browse/HDFS-11639
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs
>            Reporter: Ewan Higgs
>            Assignee: Ewan Higgs
>         Attachments: HDFS-11639-HDFS-9806.001.patch, HDFS-11639-HDFS-9806.002.patch,
HDFS-11639-HDFS-9806.003.patch
>
>
> As part of the {{PROVIDED}} storage type, we have a {{BlockAlias}} type which encodes
information about where the data comes from. i.e. URI, offset, length, and nonce value. This
data should be encoded in the protocol ({{LocatedBlockProto}} and the {{BlockTokenIdentifier}})
when a block is available using the PROVIDED storage type.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message