Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 9 Feb 2017 20:20:41 +0000 (UTC)
From: "Chris Douglas (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13013196.1476797505000.42086.1486671641856@Atlassian.JIRA>
In-Reply-To: <JIRA.13013196.1476797505000@Atlassian.JIRA>
References: <JIRA.13013196.1476797505000@Atlassian.JIRA> <JIRA.13013196.1476797505718@jira-lw-us.apache.org>
Subject: [jira] [Commented] (HDFS-11026) Convert BlockTokenIdentifier to use
 Protobuf
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 09 Feb 2017 20:20:51 -0000


    [ https://issues.apache.org/jira/browse/HDFS-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860127#comment-15860127 ] 

Chris Douglas commented on HDFS-11026:
--------------------------------------

bq. If the long (I know, I said byte) is a magic number like min or max long, decode rest as PB which will include the real expiry, else continue decoding as writable.

In #2, the magic number matches the {{field number << 3 | type}} from PB (0x08 in the current patch, IIRC). Using the {{WritableUtils}} parsing machinery is a nice touch, though we still need the marker byte at the head of the stream when we call protobuf. We're still prevented from removing the expiryTime field while we support 3.x. I suppose it can evolve to #3, where we include the expiryTime field to support 3.x clients. Implementors not targeting 2.x get a normal PB record.

What you describe is closer to #3, where the first byte is an optional PB field that we strip away before handing the record to protobuf. The protobuf parser doesn't know or care if we strip the first, optional field. It'll be vestigial in future implementations that don't support backwards compatibility with 2.x. Parsing is simplified because we don't need mark/reset (or similar), but we add a (trivial) overhead to each record and a weird compatibility case. Also the indignity of adding a version field to a DDL whose purpose was to manage versioning.

Again, we're splitting hairs, here. It's very unlikely we'll remove/replace expiryTime, or notice the overhead of an explicit version field. I'm +1 on the current patch (with a note in the .proto to help future maintainers), but would be fine with either variant.

> Convert BlockTokenIdentifier to use Protobuf
> --------------------------------------------
>
>                 Key: HDFS-11026
>                 URL: https://issues.apache.org/jira/browse/HDFS-11026
>             Project: Hadoop HDFS
>          Issue Type: Task
>          Components: hdfs, hdfs-client
>    Affects Versions: 2.9.0, 3.0.0-alpha1
>            Reporter: Ewan Higgs
>            Assignee: Ewan Higgs
>             Fix For: 3.0.0-alpha3
>
>         Attachments: blocktokenidentifier-protobuf.patch, HDFS-11026.002.patch, HDFS-11026.003.patch, HDFS-11026.004.patch, HDFS-11026.005.patch
>
>
> {{BlockTokenIdentifier}} currently uses a {{DataInput}}/{{DataOutput}} (basically a {{byte[]}}) and manual serialization to get data into and out of the encrypted buffer (in {{BlockKeyProto}}). Other TokenIdentifiers (e.g. {{ContainerTokenIdentifier}}, {{AMRMTokenIdentifier}}) use Protobuf. The {{BlockTokenIdenfitier}} should use Protobuf as well so it can be expanded more easily and will be consistent with the rest of the system.
> NB: Release of this will require a version update since 2.8.x won't be able to decipher {{BlockKeyProto.keyBytes}} from 2.8.y.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org