hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Templeton <dan...@cloudera.com>
Subject Changing the JSON Serializer
Date Thu, 16 Nov 2017 03:23:43 GMT
Looks like our REST endpoints return malformed JSON for any DAO that 
includes a Map.  That includes:

* the resourceSecondsMap and preemptedResourceSecondsMap entries in all 
the GET /apps/* endpoints,
* the operationsInfo entry in the GET /scheduler endpoint for capacity 
scheduler,
* the local_resources, environment, and acls entries in the POST /apps 
endpoint, and
* the labelsToNodes entry in the GET /label-mappings endpoint.

The issue is that each entry in the map is included with a duplicate key 
("entry").  Some JSON parsers will choke on the error, and some will 
quietly drop the duplicates.  I've filed YARN-7505 to address the issue.

The solution is to replace the Jersey JSON serializer with the Jackson 
JSON serializer.  This change fixes the issue, but it changes the 
structure of the resulting JSON.  For example, without YARN-7505, 
hitting /apps might yield JSON that contains something like:

"resourceSecondsMap":{
   "entry":{"key":"memory-mb","value":"11225"},
   "entry":{"key":"vcores","value":"5"}
   "entry":{"key":"test","value":"0"}
   "entry":{"key":"test2","value":"0"}
}

With YARN-7505, we get:

"resourceSecondsMap": {
   "test2":0,
   "test":0,
   "memory-mb":11225,
   "vcores":5
}

The first example is obviously broken, so the second one is clearly 
better, but it's structurally different.

For the GET /label-mappings endpoint, the keys of the map also have to 
be changed to simple strings because JSON doesn't allow for complex map 
keys.  So this:

"labelsToNodes":{
   "entry":{
     "key":{"name":"label1","exclusivity":"true"},
     "value":{"nodes":"localhost:63261"}
   }
}

becomes this:

"labelsToNodes":{
   "label1":{
     "nodes":["dhcp-10-16-0-181.pa.cloudera.com:63261"]
   }
}

The first one sucks and is invalid, but changing to the second one will 
break clients that are parsing the first one, especially if they're 
expecting to get the label exclusivity from this endpoint.

Before I try to get YARN-7505 committed, I want to give the community a 
chance to voice any concerns about the change.  It's too late to get 
into 3.0.0, so we'd be looking at 3.0.1 and 3.1.0.

Feel free to comment here or on the JIRA directly.

Thanks,
Daniel

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message