hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Douglas <chris.doug...@gmail.com>
Subject Re: Can we update protobuf's version on trunk?
Date Wed, 29 Mar 2017 00:09:01 GMT
On Tue, Mar 28, 2017 at 4:18 PM, Andrew Wang <andrew.wang@cloudera.com> wrote:
> Unfortunately, it sounds like these are intrinsic differences with PB3.

That's too bad... but possibly not fatal: most of the data we proxy
through client code is, if not opaque, it's at least immutable
(particularly tokens). If PB3 does support reading valid PB fields as
bytes, then we could proxy the payload through application code as an
opaque blob. That opacity has a drawback: if clients could use that
information (e.g., StorageType), we'd need to include it in a
redundant field.

Ewan Higgs used a technique in HDFS-11026 [1] to handle a transition
from Writable to protobuf. This probably could be used for most of our
token types. It's not a general solution, but it would be sufficient
for existing applications to continue working, with some accommodation
for proxy versioning and rolling upgrades.

I haven't seen data identifying PB as a bottleneck, but the
non-x86/non-Linux and dev setup arguments may make this worthwhile. -C

[1] https://issues.apache.org/jira/browse/HDFS-11026

To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

View raw message