hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milind Bhandarkar <mbhandar...@gopivotal.com>
Subject [Proposal] libyarn - a native C API for Yarn
Date Tue, 15 Oct 2013 03:09:54 GMT
Hello Folks,

As some of you might know, we have been trying to bridge the OpenMPI
community with the Hadoop community, by trying to offer a choice of YARN as
one of the  resource managers for OpenMPI. (For those who are not aware of
OpneMPI, it has a very pluggable runtime to plug-in various proprietary and
open-source components at various levels, and has been very successful due
to this pluggability.)

In order to make  YARN as a viable resource manager for the OpenMPI
ecosystem, one of the projects that we, at Pivotal, have been building for
the last 18 months, is libyarn, which is a native C interface using
protobuf-c to interact with YARN as a resource manager within the ORTE
(OpenMPI Runtime Environment). The reason this has taken so long, is
because the YARN APIs, and protocols have been changing constantly while we
were  building this. As I have stated in one of the related Jiras, this
effort has been like building a second floor on a mobile home, while it is
moving at 60MPH on the back of a huge truck (Hadoop) on a highway.

We have so far implemented most of the YARN APIs needed to get a pure
C-based YARN client (and Application Master) to exist in the upcoming
Hadoop 2.2, and nothing more, nothing less. As usual, I should caution you
though, that this is very much a work-in-progress, and not the final
product. (An example: Kerberized security in a non-java-based client still
remains to be done, but once you move the broker e.g HiveServer inside the
security perimeter, it does not matter much for actual users.)

We would like to contribute libyarn  to Apache YARN.

I would like to seek opinions of YARN developer community to see whether
libyarn has a place within YARN itself, or does it belong on github.

The reason I mentioned github, is because Arun Murthy, the creator of YARN,
recently published a go-language interface(the second non-java native yarn
interface IMHO)  to YARN, and instead of proposing it as an addition to
Apache YARN, and published it on Hortonworks github.

What does the yarn-dev community think ? Should libyarn be contributed to
Apache YARN or should it be part of our own github repo ?

-          Milind

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message