hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingyu Kim <m...@palantir.com>
Subject Securely discovering Application Master's metadata or sending a secret to Application Master at submission
Date Thu, 09 Jun 2016 05:05:49 GMT
Hi all,


To provide a bit of background, I’m trying to deploy a REST server on Application Master
and discover the randomly assigned port number securely. I can easily discover the host name
of AM through YARN REST API, but the port number needs to be discovered separately. (Port
number is assigned within a specified range with retries to avoid port conflicts) An easy
solution would be to have Application Master make a callback with the port number, but I’d
like to design it such that YARN nodes don’t talk back to the node that submitted the YARN
application. So, this problem reduces to securely discovering a small metadata of Application
Master. To be clear, by being secure, I’m less concerned about exposing the information
to others, but more concerned about the integrity of data (e.g. the metadata actually originated
from the Application Master.)


I was hoping that there is a way to register some Application Master metadata to Resource
Manager, but there doesn’t seem to be a way. Another option I considered was to write the
information to a HDFS file, but in order to verify the integrity of the content, I need a
way to securely send a private key to Application Master, which I’m not sure what the best


To recap, does anyone know if there is a way

·         To register small metadata securely from Application Master to Resource Manager
so that it can be discovered by the YARN application submitter?

·         Or, to securely send a private key to Application Master at the application submission


Thanks a lot,


View raw message