hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Templeton <dan...@cloudera.com>
Subject Re: Changing the JSON Serializer
Date Sat, 18 Nov 2017 00:49:29 GMT
Oh, yeah, I agree.  Adding the new version would be a 3.1 thing.

Daniel

On 11/17/17 4:48 PM, Sean Busbey wrote:
> I personally wouldn't be comfortable adding an API version in a maintenance
> release; it's essentially adding a feature. but I'm not set to be the RM
> for 3.0.1. :)
>
> On Nov 17, 2017 17:56, "Eric Yang" <eyang@hortonworks.com> wrote:
>
>> This means YARN-7505 can have /ws/v2/* running in parallel of /ws/v1/* for
>> 3.1 or 3.0.1 release, and deprecate /ws/v1/*.  In version 4, we drop
>> /ws/v1/*, right?
>>
>> I think this plan can work.
>>
>>
>>
>> Regards,
>>
>> Eric
>>
>>
>>
>>
>>
>> *From: *Sean Busbey <busbey@cloudera.com>
>> *Date: *Friday, November 17, 2017 at 3:08 PM
>> *To: *Eric Yang <eyang@hortonworks.com>
>> *Cc: *"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
>> *Subject: *Re: Changing the JSON Serializer
>>
>>
>>
>> 3.0.0 RCs are in progress already. Bit late to make a breaking change.
>>
>>
>>
>> the REST APIs are versioned for a reason. So long as we're outputting
>> these changes on a new version, this change should be fine on whatever
>> branch we like. When we open up for changes to go in the next major release
>> we can drop the v1 APIs.
>>
>>
>>
>> On Fri, Nov 17, 2017 at 11:41 AM, Eric Yang <eyang@hortonworks.com> wrote:
>>
>> +1 on changing the JSON serializer.  Hadoop was an early adopter for
>> Jersey, but proper JSON deserializer for Jackson didn’t appear until mid
>> 2016 after Jackson 2.5 release.  Hence, some early versions of Hadoop REST
>> API were not JSON compliant.  Hadoop kind of comply to schematic
>> versioning, therefore, it will be best to make this change in 3.0 release.
>> This will reduce some baggage carried forward from Hadoop 2.x.
>> I think community will respond positively toward this change.  Thank you
>> for bringing this up.
>>
>> regards,
>> Eric
>>
>>
>> On 11/16/17, 10:02 PM, "Sean Busbey" <busbey@cloudera.com> wrote:
>>
>>      The REST APIs are covered under the compatibility guidelines[1].
>> Presuming
>>      these are under a new API version number, it's not clear to me from the
>>      existing guidelines if adding one is okay in a maintenance release. It
>>      sounds surprising to me.
>>
>>      [1]:
>>      https://hadoop.apache.org/docs/current/hadoop-project-
>> dist/hadoop-common/Compatibility.html#REST_APIs
>>
>>      On Wed, Nov 15, 2017 at 9:23 PM, Daniel Templeton <daniel@cloudera.com
>>      wrote:
>>
>>      > Looks like our REST endpoints return malformed JSON for any DAO that
>>      > includes a Map.  That includes:
>>      >
>>      > * the resourceSecondsMap and preemptedResourceSecondsMap entries in
>> all
>>      > the GET /apps/* endpoints,
>>      > * the operationsInfo entry in the GET /scheduler endpoint for
>> capacity
>>      > scheduler,
>>      > * the local_resources, environment, and acls entries in the POST
>> /apps
>>      > endpoint, and
>>      > * the labelsToNodes entry in the GET /label-mappings endpoint.
>>      >
>>      > The issue is that each entry in the map is included with a duplicate
>> key
>>      > ("entry").  Some JSON parsers will choke on the error, and some will
>>      > quietly drop the duplicates.  I've filed YARN-7505 to address the
>> issue.
>>      >
>>      > The solution is to replace the Jersey JSON serializer with the
>> Jackson
>>      > JSON serializer.  This change fixes the issue, but it changes the
>> structure
>>      > of the resulting JSON.  For example, without YARN-7505, hitting
>> /apps might
>>      > yield JSON that contains something like:
>>      >
>>      > "resourceSecondsMap":{
>>      >   "entry":{"key":"memory-mb","value":"11225"},
>>      >   "entry":{"key":"vcores","value":"5"}
>>      >   "entry":{"key":"test","value":"0"}
>>      >   "entry":{"key":"test2","value":"0"}
>>      > }
>>      >
>>      > With YARN-7505, we get:
>>      >
>>      > "resourceSecondsMap": {
>>      >   "test2":0,
>>      >   "test":0,
>>      >   "memory-mb":11225,
>>      >   "vcores":5
>>      > }
>>      >
>>      > The first example is obviously broken, so the second one is clearly
>>      > better, but it's structurally different.
>>      >
>>      > For the GET /label-mappings endpoint, the keys of the map also have
>> to be
>>      > changed to simple strings because JSON doesn't allow for complex map
>> keys.
>>      > So this:
>>      >
>>      > "labelsToNodes":{
>>      >   "entry":{
>>      >     "key":{"name":"label1","exclusivity":"true"},
>>      >     "value":{"nodes":"localhost:63261"}
>>      >   }
>>      > }
>>      >
>>      > becomes this:
>>      >
>>      > "labelsToNodes":{
>>      >   "label1":{
>>      >     "nodes":["dhcp-10-16-0-181.pa.cloudera.com:63261"]
>>      >   }
>>      > }
>>      >
>>      > The first one sucks and is invalid, but changing to the second one
>> will
>>      > break clients that are parsing the first one, especially if they're
>>      > expecting to get the label exclusivity from this endpoint.
>>      >
>>      > Before I try to get YARN-7505 committed, I want to give the
>> community a
>>      > chance to voice any concerns about the change.  It's too late to get
>> into
>>      > 3.0.0, so we'd be looking at 3.0.1 and 3.1.0.
>>      >
>>      > Feel free to comment here or on the JIRA directly.
>>      >
>>      > Thanks,
>>      > Daniel
>>      >
>>      > ------------------------------------------------------------
>> ---------
>>      > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>      > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>      >
>>      >
>>
>>
>>      --
>>      busbey
>>
>>
>>
>>
>>
>> --
>>
>> busbey
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message