hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Trivial Update of "Hbase/Plan-0.17" by stack
Date Thu, 07 Feb 2008 04:51:26 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by stack:

The comment on the change is:

  == Plan for hbase 0.17.0 version ==
+ A meeting to plan hbase development priorities over the next three months, "hbase 0.17",
took place January 30th, 2008 at Powerset in San Francisco.  Attendees were Chris Kline, Bryan
Duxbury, Jim Firby, Michael Bieniosek, Jim Kellerman, Chad Walters, and Michael Stack.
- Meeting took place January 30th, 2008 at Powerset in San Francisco.
+ Format was a round of the room giving each of the attendees opportunity to call out the
big ticket items that they believed needs addressing over the next phase of development. 
Thereafter, the meeting degenerated into a huddle around a screen with Bryan, Jim K, and stack
entering and jumbling issues to match the agreed upon priorities.
- Robustness and Scalability Release
+ High level, the consensus was that the focus for the next three months should be '''Robustness
and Scalability''' (In that order).
- Commit to maintaining 0.16.0; we won't abandon it like we did 0.15.x ten minutes after it
was released
+ === Identified Priorities ===
+ Apart from the obvious hadoop issues -- i.e. HADOOP-1700 -- the group identified the below
as priorities to be addressed over the course of the next three months or so. All items make
contribution to the banner target of increased "Robustness and Scalability".  All items have
by now corresponding issues over in JIRA.
- High level:
+  * "Too many open file handles":  Fix the hbase file handle gluttony.
+  * Need to add an accounting of hbase resource usage -- e.g. A Memory Manager -- and we
need to add means of having the regionserver tell the master "I'm full" and for the master's
part, it should not try assigning any more regions to this server.
+  * Assignment functions needs work. Fix it so one server sometimes gets 100s of regions
while others get two or three only.  Also, fix startup sequence.  Currently if many regions
to assign, there is churn with regions multiply assigned (Usually it settles eventually, but
fix the churn).
+  * Rebalancing load.
+  * Make start/stop reliable.  Currently, regionserver goes down though it never talked to
the master.  Should stay up.  Also, hbase should be able to work in absence of hdfs -- or
at least not freeze when its pulled out from under hbase -- and it also needs to be start/stop
nicely (as best as it can) independent of the order in which applications are started: e.g.
if hbase is started before hdfs, hbase should just gracefully hang till hdfs appears and then
continue on its way
+  * Means of making the cluser read-only/freeze-it
+  * Means of checkpointing/syncing (Perhaps based-off or labelled by timestamp).
+  * We've seen hbase corrupt HDFS.  Figure out how.
- "Too many open file handles"  Priority.  Scalability.
+ Target is that the next release can handle "3TB of data on about ~50 nodes".
- Caching.  Low priority.
+ === Other items discussed ===
+  * hbase needs a wikipedia page.
+  * We should make an hbase blog.  It would replace the 'news' section up on hbase wiki
+  * Remove deprecated methods.
+  * Caching came up but was thought a low priority though it was allowed that since Tom White's
work, would take little to add a caching of "hot rows".
+  * We took a vote and HBase (capital B) overwhelmingly beat Hbase (little b) as way to capitalize
name of this project
- Memory Manager.  Priority.  "I'm full"
+ === Refactoring Patterns ===
+ There was some talk among J, B, and S about refactoring patterns to keep in mind as we hack
into the hbase future: 
- Assignment functions.  Needs work.  So one server doesn't get 100s while others get two
or three.  Startup sequence: currently churn when 100s of regions multiply assigning regions.
+  * Make new subpackages o.a.h.h.regionserver, o.a.h.h.master, and o.a.h.h.client
+  * Move inner classes < 20 lines or so out of containing classes into classes of their
own.  If master, regionserver or client inner classes, new classes should be package private,
+  * Take on the Callable pattern introduced by Peter Dolan.  There's a load of places where
it can be used to get rid of duplicated retry code.
- Rebalancing load.
- Start/Stop reliable.  Regionserver goes down though its never talked to master.  Needs to
work independent of hdfs and order.
- Read-only/Freeze
- Checkpointing/Sync/Timestamp.
- Blog, hbase wikipedia page.
- 3TB and about ~50 nodes.
- Integration tests rather than unit tests.  Then verification tests.  Mock objects.
- Remove deprecated methods.
- Corrupt HDFS.
- Refactoring patterns.
- Move o.a.h.h.regionserver, master, client
- Callable pattern
- Inner classes no unless < 20 lines  (Pkg private)
- Lets have a meeting, those of us who are near-by, and agree what we hope to achieve over
the next three months in time for the 0.17 release of hbase.
- Here's work I know we need to do in 0.17 timeframe:
-  * Move out from under hadoop moving hbase to its own svn respository, develop hbase website,
- Here's some proposals to consider when we meet.
-  * Propose that we make a commitment to keep up a stable 0.16.x version of hbase backporting
critical fixes.  Folks are beginning to depend on hbase.  They need to be shielded from the
churn in TRUNK.  Let them run 0.16.x and we'll support them.
-  * Propose we agree on a couple of refactoring vectors -- e.g. move regionserver and master
into servers subpackage or each to their own package then call a mass eviction on the bloating
inner classes -- and refactoring patterns to apply where we see fit: e.g. Pattern used in
HADOOP-255 refactoring of HTable gets.
-  * Propose review of JIRAs prioritizing and slating appropriate to a 0.17 release
- Anything else? Is it Hbase or HBase?
- ''From Michael Bieniosek:''
- I have some feature requests:
-  * deal with DFS too-many-file-handles problem: maybe take inactive regions offline, or
at least close unused DFS file handles?
-  * region mirroring (for hot regions, and for dealing with slow regionservers)
-  * marking tables read-only (for avoiding accidental modifications, and as a possible optimization
enabler) -- (This is HADOOP-1958)

View raw message