hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11026) Convert BlockTokenIdentifier to use Protobuf
Date Thu, 09 Feb 2017 20:20:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15860127#comment-15860127

Chris Douglas commented on HDFS-11026:

bq. If the long (I know, I said byte) is a magic number like min or max long, decode rest
as PB which will include the real expiry, else continue decoding as writable.

In #2, the magic number matches the {{field number << 3 | type}} from PB (0x08 in the
current patch, IIRC). Using the {{WritableUtils}} parsing machinery is a nice touch, though
we still need the marker byte at the head of the stream when we call protobuf. We're still
prevented from removing the expiryTime field while we support 3.x. I suppose it can evolve
to #3, where we include the expiryTime field to support 3.x clients. Implementors not targeting
2.x get a normal PB record.

What you describe is closer to #3, where the first byte is an optional PB field that we strip
away before handing the record to protobuf. The protobuf parser doesn't know or care if we
strip the first, optional field. It'll be vestigial in future implementations that don't support
backwards compatibility with 2.x. Parsing is simplified because we don't need mark/reset (or
similar), but we add a (trivial) overhead to each record and a weird compatibility case. Also
the indignity of adding a version field to a DDL whose purpose was to manage versioning.

Again, we're splitting hairs, here. It's very unlikely we'll remove/replace expiryTime, or
notice the overhead of an explicit version field. I'm +1 on the current patch (with a note
in the .proto to help future maintainers), but would be fine with either variant.

> Convert BlockTokenIdentifier to use Protobuf
> --------------------------------------------
>                 Key: HDFS-11026
>                 URL: https://issues.apache.org/jira/browse/HDFS-11026
>             Project: Hadoop HDFS
>          Issue Type: Task
>          Components: hdfs, hdfs-client
>    Affects Versions: 2.9.0, 3.0.0-alpha1
>            Reporter: Ewan Higgs
>            Assignee: Ewan Higgs
>             Fix For: 3.0.0-alpha3
>         Attachments: blocktokenidentifier-protobuf.patch, HDFS-11026.002.patch, HDFS-11026.003.patch,
HDFS-11026.004.patch, HDFS-11026.005.patch
> {{BlockTokenIdentifier}} currently uses a {{DataInput}}/{{DataOutput}} (basically a {{byte[]}})
and manual serialization to get data into and out of the encrypted buffer (in {{BlockKeyProto}}).
Other TokenIdentifiers (e.g. {{ContainerTokenIdentifier}}, {{AMRMTokenIdentifier}}) use Protobuf.
The {{BlockTokenIdenfitier}} should use Protobuf as well so it can be expanded more easily
and will be consistent with the rest of the system.
> NB: Release of this will require a version update since 2.8.x won't be able to decipher
{{BlockKeyProto.keyBytes}} from 2.8.y.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message