hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ewan Higgs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11026) Convert BlockTokenIdentifier to use Protobuf
Date Thu, 03 Nov 2016 08:30:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15632055#comment-15632055
] 

Ewan Higgs commented on HDFS-11026:
-----------------------------------

{quote}
Are you saying you can't make your patch work in a compatible way? Or you can't get stock
2.8/3.0 to work together?
{quote}

The latter. When I try to start NN and DN across 2.8/3.0 boundaries I get the following error
(NN 2.8, DN 3.0):
{code}
2016-11-01 11:35:25,727 WARN datanode.DataNode: The reported NameNode version is too low to
communicate with this DataNode. NameNode version: '2.8.0-SNAPSHOT' Minimum NameNode version:
'3.0.0-alpha1-SNAPSHOT'
2016-11-01 11:35:25,728 ERROR datanode.DataNode: Initialization failed for Block pool <registering>
(Datanode Uuid unassigned) service to localhost/127.0.0.1:60020. Exiting. 
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: The reported NameNode version
is too low to communicate with this DataNode. NameNode version: '2.8.0-SNAPSHOT' Minimum NameNode
version: '3.0.0-alpha1-SNAPSHOT'
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.checkNNVersion(BPServiceActor.java:252)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:239)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:271)
	at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:782)
	at java.lang.Thread.run(Thread.java:745)
{code}

or NN 3.0, DN 2.8:

{code}
2016-11-01 11:30:26,767 INFO datanode.DataNode: Block pool BP-875416690-10.108.37.99-1476803009605
(Datanode Uuid 39b7e605-d1ec-470e-a4e5-39b0c0d4bd9f) service to localhost/127.0.0.1:60020
beginning handshake with NN
2016-11-01 11:30:26,818 ERROR datanode.DataNode: Initialization failed for Block pool BP-875416690-10.108.37.99-1476803009605
(Datanode Uuid 39b7e605-d1ec-470e-a4e5-39b0c0d4bd9f) service to localhost/127.0.0.1:60020
The reported DataNode version is too low to communicate with this NameNode. DataNode version:
'2.8.0-SNAPSHOT' Minimum DataNode version: '3.0.0-alpha1-SNAPSHOT'
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.verifySoftwareVersion(NameNodeRpcServer.java:1663)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.registerDatanode(NameNodeRpcServer.java:1402)
	at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.registerDatanode(DatanodeProtocolServerSideTranslatorPB.java:100)
	at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:30055)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:467)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:990)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
	at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1795)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2535)
{code}

> Convert BlockTokenIdentifier to use Protobuf
> --------------------------------------------
>
>                 Key: HDFS-11026
>                 URL: https://issues.apache.org/jira/browse/HDFS-11026
>             Project: Hadoop HDFS
>          Issue Type: Task
>          Components: hdfs, hdfs-client
>    Affects Versions: 2.9.0, 3.0.0-alpha1
>            Reporter: Ewan Higgs
>             Fix For: 3.0.0-alpha2
>
>         Attachments: HDFS-11026.002.patch, blocktokenidentifier-protobuf.patch
>
>
> {{BlockTokenIdentifier}} currently uses a {{DataInput}}/{{DataOutput}} (basically a {{byte[]}})
and manual serialization to get data into and out of the encrypted buffer (in {{BlockKeyProto}}).
Other TokenIdentifiers (e.g. {{ContainerTokenIdentifier}}, {{AMRMTokenIdentifier}}) use Protobuf.
The {{BlockTokenIdenfitier}} should use Protobuf as well so it can be expanded more easily
and will be consistent with the rest of the system.
> NB: Release of this will require a version update since 2.8.x won't be able to decipher
{{BlockKeyProto.keyBytes}} from 2.8.y.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message