hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Douglas <chris.doug...@gmail.com>
Subject Re: Can we update protobuf's version on trunk?
Date Tue, 28 Mar 2017 23:04:11 GMT
On Tue, Mar 28, 2017 at 3:04 PM, Andrew Wang <andrew.wang@cloudera.com> wrote:
> There's no mention of the convenient "Embedded messages are compatible with
>> bytes if the bytes contain an encoded version of the message" semantics in
>> proto3.
>
>
> I checked the proto3 guide, and I think this is supported:
> https://developers.google.com/protocol-buffers/docs/proto3#updating

You're right, it looks like this is supported.

> If unknown fields are dropped, then applications proxying tokens and other
>> data between servers will effectively corrupt those messages, unless we
>> make everything opaque bytes, which- absent the convenient, prenominate
>> semantics managing the conversion- obviate the compatibility machinery that
>> is the whole point of PB. Google is removing the features that justified
>> choosing PB over its alternatives. Since we can't require that our
>> applications compile (or link) against our updated schema, this creates a
>> problem that PB was supposed to solve.
>
>
> This is scary, and it potentially affects services outside of the Hadoop
> codebase. This makes it difficult to assess the impact.

Stack mentioned a compatibility mode that uses the proto2 semantics.
If that carries unknown fields through intermediate handlers, then
this objection goes away. -C

> Paraphrasing, the issues with PB2.5 are:
>
>    1. poor support for non-x86, non-Linux platforms
>    2. not as available, so harder to setup a dev environment
>    3. missing zero-copy support, which helped performance in HBase
>
> #1 and #2 can be addressed if we rehosted PB (with cross-OS compilation
> patches) elsewhere.
> #3 I don't think we benefit from, since we don't pass around big PB byte
> arrays (at least in HDFS).
>
> So the way I see it, upgrading to PB3 has risk from the behavior change wrt
> unknown fields, while there are other ways of addressing the stated issues
> with PB2.5.
>
> Best,
> Andrew

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message