hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Seth <seth.siddha...@gmail.com>
Subject Re: serializing/deserializing wrapped protobuf generated classes
Date Tue, 21 May 2013 07:48:27 GMT
I'd agree. Using ProtoBase in application code is not a good idea.

The MR AM currently uses the string form of the IDs to write to the history
file, and constructs them back from this. This would likely not be
practical for some of the more complicated records like Container.

I initially thought a serialization/deserialization interface on all
records would be useful (especially in the context of RM state store), but
have since been convinced otherwise - YARN does not need to expose it's
underlying record serialization mechanism to user code - not via the
individual records itself.  That said, should YARN be providing a utility
library to help app developers with this functionality. This would end up
becoming part of the public API, along with the data format. What do others
think ?

- Sid

On Mon, May 20, 2013 at 11:04 AM, Alejandro Abdelnur <tucu@cloudera.com>wrote:

> [I know it has been discussed before the need (or not) of having the
> current wrappers hiding the protobuf generated classes].
> I order to do things like AMs failover and checkpointing I need to
> serialize app IDs, app attempt IDs, containers and/or IDs,  resource
> requests, etc.
> This means that the current wrapping hides the PB impl, thus hiding the
> provided ser/deser capabilities.
> I could force-cast a record to ProtoBase (which is private) and then get
> the PROTO Message and then do the ser/deser with that.
> But this, IMO, is a no no.
> Thoughts?
> Thanks
> --
> Alejandro

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message