hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Large feature development
Date Sat, 01 Sep 2012 19:47:51 GMT

On Sep 1, 2012, at 1:20 AM, Todd Lipcon wrote:

> I'd actually contend that YARN was merged too early. I have yet to see
> anyone running YARN in production, and it's holding up the "Stable"
> moniker for Hadoop 2.0 -- HDFS-wise we are already quite stable and
> I'm seeing fewer issues in our customers running Hadoop HDFS 2
> compared to Hadoop 1-derived code.

You know I respect you a ton, but I'm very saddened to see you perpetuate this FUD on our
public lists. I expected better, particularly when everyone is working towards the same goals
of advancing Hadoop-2. This sniping on other members doing work is, um, I'll just stop here
rather than regret later.

I'm pretty sure you realize this (we've talked about this privately), yet, for other users
who might not be aware:
# YARN has been deployed on, by almost everyone's standards, on a very LARGE ~450 node cluster
for 6 months now at Yahoo.
# The entire YARN & MapReduce developer community has done an enormous amount of testing,
compatibility work and performance work for many months now. It's been clear that YARN/MRv2
is superior to MR1 on every dimension - performance (2x in several cases), scale etc.; all
dimensions which are critical for Hadoop's success in the past and future.
# Not just MR, this work has been done across the stack - Pig, Oozie, HCatalog etc. This has
been an enormous amount of work not just by YARN/MRv2, but by all these communities.
# Many thousands of unique end-user applications at Yahoo have *certified* YARN/MRv2. That
is pretty much *all* MapReduce, Pig etc. applications at Yahoo - the most advanced Hadoop
deployment in the world.
# It is now *days* away from being deployed on one of the largest and most demanding Hadoop
clusters in the world with several *thousand* nodes and millions of applications per month.
See Bobby's note if you don't believe me.

Notice, I didn't talk about any of the other benefits of YARN such as other frameworks to
MR etc. - you'll see more of this such as real-time applications on Hadoop clusters over the
next many months. For e.g. see discussions on Storm/S4 lists about YARN prototypes at various
stages of availability.

Paying you back with the same coin, after being declared *done*, HDFS2 had several BASIC issues
such as a non-working upgrade from hadoop-1 (HDFS-3731, HDFS-3579) or edit-log corruption
(HDFS-3626). Maybe you or the customers you talk about don't care about it, whatever. For
e.g. is the QJM work part of stable HDFS2? It's not even code complete yet. 

IAC, It's pretty obvious we have different standards for declaring HDFS stable v/s YARN/MRv2
as stable. The standards I'm used to, being around since the dawn of this project, is what
I use to measure stability i.e. deployed and stable for weeks/months on some of the largest
Hadoop clusters in the world before letting it loose on other 'customers'. 

Given that upgrade-failures or data-corruption is acceptable, is YARN 'stable'? By the same
standards - YES! - for many months now, much before HDFS HA was even code complete!

I don't want to engage in a debate on this further or expect you to care about YARN/MRv2,
but please, for heavens' sake, do not publicly diss the work so many people have done for
many, many months now or accuse them of *holding up Hadoop* - it's very poor form. 

I'm very proud to have contributed to this effort, even more to have worked with such a talented
and dedicated bunch. A acknowledgement would be nice, but the least I/we *do* expect is absence
of public sniping by other members of the Hadoop community.


View raw message