hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Kellerman (POWERSET)" <Jim.Keller...@microsoft.com>
Subject You guys rocked the house this week!
Date Sun, 14 Jun 2009 10:03:01 GMT
I have received nothing but compliments in all my schmoozing
this week.

Although I was mostly absent from 0.20, it is 0.20
that has everyone excited. Congrats, and great work guys.

However, we still have to deliver on 0.20. It has to be rock
solid, or the buzz will turn against us.

Friday, I was at Cloudera along with Doug C, Erik14, Owen O'Malley,
Arun Murthy, Alan Gates (Pig), a guy from Hive (whose name I
can't remember at the moment), Dhruba (facebook) and, of course,
the Cloudera guys (Todd Lipcon, Jeff Hammerbacher, Christopne,
Amr, etc.)

The day went something like this:

1. 1st exercise: write (on a postit) 5 things you like about
   hadoop and 5 things you don't - most people submitted more
   than 10) and discussion.

2. 2nd exercise: write (on a postit) features that you'd like
   to see in the short term in hadoop

   - We had submissions that were truly short term and some that
     were truly "blue sky". These were divided into categories:
     Map/Reduce, HDFS, Build/Test, Core (including Avro)

   - We then split up into separate sessions. I attended HDFS.
     (the session leaders are supposed to send in notes from
     their session, and as soon as I get them, I will post.)

   - The biggest issue from HDFS was append (actually flush/sync),
     and not just me, there were about 7 votes for it (just "append")
     whereas my votes were like "flush/sync in 0.21" and HADOOP-4379
     in 0.20.x.

3. Third session: Blue Sky: not much happened here because every
   one was kind of burned out at this point.

Important points (for HBase):

1. We need to deliver a rock-solid 0.20 release or we will lose
   all the credibility that we gained this week.

   BTW: Cloudera's next release is going to be based on 0.20, and
   they will either include HBase as alpha software, or put us
   in their supported stack, depending on the reaction from our
   community. And despite the fact that their revenue stream depends
   on the Hadoop community, I got the feeling that they are getting
   pressured to have a version of HBase (not so much on 0.18, but
   more on 0.20). They have a '$' interest in seeing us succeed.

2. Once we get 0.20 out, we need to focus on beating the sh*t out
   of HADOOP-4379 patch for 0.20. Once we think it is solid, we
   need to create a script that randomly fails region server and
   datanodes. If we do this, Cloudera has volunteered to run that
   script on EC2 on a ~100 node cluster to burn it in. (They have
   some arrangement with Amazon) and they have volunteered to run
   the test on a "big" cluster for us. They will run it for several
   days if necessary to prove that it works.

   -- We need to be sure HADOOP-4379 is solid, which could lead to
      getting 4379 into hadoop 0.20.x if so. Dhruba, who led the HFDS
      breakout session, will do what it takes to fix issues around
      his current patch, provided we provide feedback to him. However,
      if we don't do #1 above, it won't matter.

   -- Master failover works

   -- Region server failover works.

3. After this week, both PIG and Hive are excited about using
   HBase as a source and a sink for their map-reduce jobs that
   they spawn. They have both come to realize that we are
   becoming more important in the Hadoop community, and are
   willing to devote resources to make their stuff work with HBase.
   (They will look bad if the other supports HBase and they do
   not - although to be fair, there was no data store available
   before this that met their needs either).

So keep up the great work and MAKE SURE 0.20 IS ROCK SOLID STABLE!!!

Jim Kellerman, Powerset (Live Search, Microsoft Corporation)

View raw message