hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-913) Add a way to register long-lived services in a YARN cluster
Date Fri, 04 Oct 2013 16:29:45 GMT

    [ https://issues.apache.org/jira/browse/YARN-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13786299#comment-13786299

Robert Joseph Evans commented on YARN-913:

Yes it does have plenty of races.  I'll try to get some detailed designs up shortly but at
a high level the general idea is to have a restful web service.  For the most common use case
there just needs to be two interfaces.

  - Register/Monitor a Service
  - Query for Services

Part of the reason we need the service registry is to securely verify that a client is talking
to the real service, and no one has grabbed the service's port after it registered.  To do
that I want to have the concept of a verified service.  For that we would need an admin interface
for adding, updating, and removing verified services.

The registry would provide a number of pluggable ways for services to authenticate.  Part
of adding a verified service would include indicating which authentication models the service
can use to register and which "users" are allowed to register that service.

The registry could also act like a trusted Certificate Authority.  Another part of adding
in a verified service would include indicating how clients could verify they are talking to
the true service.  This could include just publishing an application id so the client can
go to the RM and get a delegation token. Another option would be having the service generate
a public/private key pair.  When the service registers it would get the private key and the
public key would be available through the discovery interface.

The plan is to also have the registry monitor the service similar to ZK.  The service would
heartbeat in to the registry periodically (could be on the order of mins depending on the
service) after a certain period of time of inactivity the service would be removed from the
registry. Perhaps we should add in an explicit unregister as well.  

I want to make sure that the data model it is generic enough that we could support something
like a web service on the gird where each server can register itself and all of them would
show up in the registry, so a service could have one or more servers that are a part of it,
and each server could have some separate metadata about it.

I also want to have a plug-in interface for discovery, so we could potentially make the registry
look like a DNS server or an SSL Certificate Authority which would make compatibility with
existing applications and clients a lot simpler. 

> Add a way to register long-lived services in a YARN cluster
> -----------------------------------------------------------
>                 Key: YARN-913
>                 URL: https://issues.apache.org/jira/browse/YARN-913
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: api
>    Affects Versions: 3.0.0
>            Reporter: Steve Loughran
>            Assignee: Robert Joseph Evans
> In a YARN cluster you can't predict where services will come up -or on what ports. The
services need to work those things out as they come up and then publish them somewhere.
> Applications need to be able to find the service instance they are to bond to -and not
any others in the cluster.
> Some kind of service registry -in the RM, in ZK, could do this. If the RM held the write
access to the ZK nodes, it would be more secure than having apps register with ZK themselves.

This message was sent by Atlassian JIRA

View raw message