hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2678) Recommended improvements to Yarn Registry
Date Thu, 16 Oct 2014 16:59:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173945#comment-14173945
] 

Steve Loughran commented on YARN-2678:
--------------------------------------

Gour: I'll do these, as they seem goood and they need to be done before the registry ships
with hadoop.

w.r.t the header in the ZK nodes, it's to deal with the problems
* znodes with 0 bytes of data still have a stated size of 12 bytes. Requiring 16 bytes of
header makes it trivial to decide whether or not it has data (makes enuming child records
faster)
* allows for future expansion to have different record types.

Here's what I propose
# drop the header
# add a {{type}} field to the json
# mandate a service record type declaration
{code}
"type":"ServiceRecord-1.0.0"
{code}
# declare that the presence of the byte sequence {{"ServiceRecord-1.0.0"}} implies the entry
is a sevice record. JSON is UTF-8 encoded, so this matches the value of the {{type}} field.
# if the string is present, the entry MUST be parseable as a service record
# declare that the absence of the sequence implies that there is no service record there.
{code}

Parsing/validating becomes one of
{code}
 if len(data)<len("ServiceRecord-1.0.0"):  raise NotFound
 if not contains(data,"ServiceRecord-1.0.0"):  raise NotFound
 if not parse(data) raise InvalidRecord
 if not valid(parse(data)) raise InvalidRecord
{code}

That is, if there isn't the string, it is not a parse error, it is simply "not a record"

Validation becomes
# assert presence of {{"type":"ServiceRecord-1.0.0"}}
# forall endpoints, {{valid(endpoint)}}
# endpoints are valid if they follow the structure, al elements in the dictionary of an address
are simple strings, etc.
that's it: all other fields are optional

I'll update the .tla file and the code to match

> Recommended improvements to Yarn Registry
> -----------------------------------------
>
>                 Key: YARN-2678
>                 URL: https://issues.apache.org/jira/browse/YARN-2678
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, resourcemanager
>            Reporter: Gour Saha
>            Assignee: Steve Loughran
>
> In the process of binding to Slider AM from Slider agent python code here are some of
the items I stumbled upon and would recommend as improvements.
> This is how the Slider's registry looks today -
> {noformat}
> jsonservicerec{
>   "description" : "Slider Application Master",
>   "external" : [ {
>     "api" : "org.apache.slider.appmaster",
>     "addressType" : "host/port",
>     "protocolType" : "hadoop/protobuf",
>     "addresses" : [ [ "c6408.ambari.apache.org", "34837" ] ]
>   }, {
>     "api" : "org.apache.http.UI",
>     "addressType" : "uri",
>     "protocolType" : "webui",
>     "addresses" : [ [ "http://c6408.ambari.apache.org:43314" ] ]
>   }, {
>     "api" : "org.apache.slider.management",
>     "addressType" : "uri",
>     "protocolType" : "REST",
>     "addresses" : [ [ "http://c6408.ambari.apache.org:43314/ws/v1/slider/mgmt" ] ]
>   }, {
>     "api" : "org.apache.slider.publisher",
>     "addressType" : "uri",
>     "protocolType" : "REST",
>     "addresses" : [ [ "http://c6408.ambari.apache.org:43314/ws/v1/slider/publisher" ]
]
>   }, {
>     "api" : "org.apache.slider.registry",
>     "addressType" : "uri",
>     "protocolType" : "REST",
>     "addresses" : [ [ "http://c6408.ambari.apache.org:43314/ws/v1/slider/registry" ]
]
>   }, {
>     "api" : "org.apache.slider.publisher.configurations",
>     "addressType" : "uri",
>     "protocolType" : "REST",
>     "addresses" : [ [ "http://c6408.ambari.apache.org:43314/ws/v1/slider/publisher/slider"
] ]
>   } ],
>   "internal" : [ {
>     "api" : "org.apache.slider.agents.secure",
>     "addressType" : "uri",
>     "protocolType" : "REST",
>     "addresses" : [ [ "https://c6408.ambari.apache.org:46958/ws/v1/slider/agents" ] ]
>   }, {
>     "api" : "org.apache.slider.agents.oneway",
>     "addressType" : "uri",
>     "protocolType" : "REST",
>     "addresses" : [ [ "https://c6408.ambari.apache.org:57513/ws/v1/slider/agents" ] ]
>   } ],
>   "yarn:persistence" : "application",
>   "yarn:id" : "application_1412974695267_0015"
> }
> {noformat}
> Recommendations:
> 1. I would suggest to either remove the string {color:red}jsonservicerec{color} or if
it is desirable to have a non-null data at all times then loop the string into the json structure
as a top-level attribute to ensure that the registry data is always a valid json document.

> 2. The {color:red}addresses{color} attribute is currently a list of list. I would recommend
to convert it to a list of dictionary objects. In the dictionary object it would be nice to
have the host and port portions of objects of addressType "uri" as separate key-value pairs
to avoid parsing on the client side. The URI should also be retained as a key say "uri" to
avoid clients trying to generate it by concatenating host, port, resource-path, etc. Here
is a proposed structure -
> {noformat}
> {
>   ...
>   "internal" : [ {
>     "api" : "org.apache.slider.agents.secure",
>     "addressType" : "uri",
>     "protocolType" : "REST",
>     "addresses" : [ 
>        { "uri" : "https://c6408.ambari.apache.org:46958/ws/v1/slider/agents",
>          "host" : "c6408.ambari.apache.org",
>          "port": "46958"
>        }
>     ]
>   } 
>   ],
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message