hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11026) Convert BlockTokenIdentifier to use Protobuf
Date Wed, 19 Oct 2016 16:30:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589177#comment-15589177

Daryn Sharp commented on HDFS-11026:

IMHO, a major upgrade shouldn't be a sufficient reason to introduce avoidable protocol incompatibilities.
 It'll guarantee that 3.x is DOA for large production deployments.

Yarn isn't exactly a good counter-example.  As you point out, yarn made the change to support
RU so the cost/benefit was justified.  What is the compelling case that breaks the crucial
feature that justified the yarn incompatibility?

The main difference between hdfs and yarn is scope of impact.  The impact from the yarn incompatibility
was confined to the cluster being upgraded.  This hdfs incompatibility would impact clients
on other clusters.  Imagine trying to get the stars to align with customers across many clusters
to give a green light for their SLAs possibly being impacted.  Not once, but every time you
upgrade dozens of clusters.

In the end, the DN will have to support dual decoding for inter-cluster clients, super long
running clients holding old tokens, the balancer, etc.  If we go forward with this change,
I'd like/prefer to see a designated 2.x "minimum release" before a 3.x upgrade.  That designated
release would add the latent support for PB tokens.

> Convert BlockTokenIdentifier to use Protobuf
> --------------------------------------------
>                 Key: HDFS-11026
>                 URL: https://issues.apache.org/jira/browse/HDFS-11026
>             Project: Hadoop HDFS
>          Issue Type: Task
>          Components: hdfs, hdfs-client
>    Affects Versions: 2.9.0, 3.0.0-alpha1
>            Reporter: Ewan Higgs
>             Fix For: 3.0.0-alpha2
>         Attachments: blocktokenidentifier-protobuf.patch
> {{BlockTokenIdentifier}} currently uses a {{DataInput}}/{{DataOutput}} (basically a {{byte[]}})
and manual serialization to get data into and out of the encrypted buffer (in {{BlockKeyProto}}).
Other TokenIdentifiers (e.g. {{ContainerTokenIdentifier}}, {{AMRMTokenIdentifier}}) use Protobuf.
The {{BlockTokenIdenfitier}} should use Protobuf as well so it can be expanded more easily
and will be consistent with the rest of the system.
> NB: Release of this will require a version update since 2.8.x won't be able to decipher
{{BlockKeyProto.keyBytes}} from 2.8.y.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message