hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Clampffer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-10595) libhdfs++: Client Name Protobuf Error
Date Mon, 11 Jul 2016 19:54:10 GMT

    [ https://issues.apache.org/jira/browse/HDFS-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15371494#comment-15371494
] 

James Clampffer commented on HDFS-10595:
----------------------------------------

I think this is a duplicate of HDFS-9453, different source but same issue.  From what I gathered
looking through the C++ protobuf source if there are embedded nulls it can fall back to a
 reflection based parser (slower).  I'm not sure if the java protobuf implementation has less
restrictive rules on string fields. 

> libhdfs++: Client Name Protobuf Error
> -------------------------------------
>
>                 Key: HDFS-10595
>                 URL: https://issues.apache.org/jira/browse/HDFS-10595
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>            Reporter: Anatoli Shein
>
> When running a cat tool (/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c)
I get the following error:
> [libprotobuf ERROR google/protobuf/wire_format.cc:1053] String field contains invalid
UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send
raw bytes.
> However it executes correctly. Looks like this error happens when trying to serialize
Client name in ClientOperationHeaderProto::SerializeWithCachedSizes (/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/datatransfer.pb.cc)
> Possibly the problem is caused by generating client name as a UUID in GetRandomClientName
(/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc)
> In Java client it looks like there are two different unique client identifiers: ClientName
and ClientId:
> Client name is generated as:
> clientName = "DFSClient_" + dfsClientConf.getTaskId() + "_" + ThreadLocalRandom.current().nextInt()
 + "_" + Thread.currentThread().getId(); (/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java)
> ClientId is generated as a UUID in (/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientId.java)
> In libhdfs++ we need to possibly also have two unique client identifiers, or fix the
current client name to work without protobuf warnings/errors.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message