hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Some notes from the HBase developer Pow-Wow
Date Wed, 20 Feb 2013 02:51:21 GMT
Sorry I couldn't make it up for the powwow.

Thanks Devaraj for the excellent notes.

On this part:

    -- Branching on a big feature was suggested -
       --- Issues: committer needs to be there
             Sergey: If the rate of change is high on common code (to
the branches)
then merging will be tough
       --- Jon: Refactor should be done in the main branch (since it doesn't
add any new funtionality)

I like how snapshots was branched and worked on collaboratively up in

It's pretty painless to rebase with git but painful to keep a branch up to
date with subversion then merge back (at least for me).

I have something in the works that has to be out of tree for a while
because it depends on new APIs for Hadoop Common. Was thinking it should go
up into GH in the meantime.

Makes sense for big features to go on a branch and then be subject to the 3
+1 rule for merge. More eyes on the change. Also makes sense for refactors
to be done in place. They may cause pain for work on a branch, but it's
less painful to merge in as it happens on the main branch than sort out
collisions later if a refactor and feature want to go in at about the same

And on this part:

    - A bunch of discussions on the RPC with KeyValue/Cells

.... / Tags :-)

On Tue, Feb 19, 2013 at 5:27 PM, Devaraj Das <ddas@hortonworks.com> wrote:

> - Stabilizing 0.96 / CI
>   -- not running continuously
>   -- Roman:   is it a good idea to run IT under Bigtop
>        - the HBase unit tests are good..
>        - What about HBase as a backend for hive - tests for those
>        - Do we think there is value with the integration with the rest
> of the hadoop ecosystem
>   -- Jon: What's needed to make this work
>   -- Roman: Collective agreement that we want to solve this
>   -- Stack: We need to run the IT tests from Enis and Sergey running
> continuously
>   -- Roman: If we all agree that this is something needs to be fixed
> .. then yes we would talk about mechanics. Bigtop doesn't have
> expertise in HBase and hence HBase folks would have to debug failures.
>   -- Roman: Bigtop is committed to tests. Less than a dozen tests for
> hbase currently..
>   -- We have been running validation around RC time. Find all kinds of
> issues - sometimes trivial (maybe a config issue),
>   -- Roman can offer CI for trunk but will work for only hadoop-2 line.
>   -- Roman does first line of triage.
>   -- No issues to do with other ecosystem artifacts. Bigtop ensures
> the right artifacts are in place.
>   -- hadoop-2 is important but not particular about the version of
> hadoop within 2. For example 2.0.2
>   -- Gary: Will be good to run security tests
>   -- Roman & Devaraj to talk on how this can be done/implemented
> On 0.96 branching
> - Lars:
>    -- We will have three branches to maintain.
> - Stack: we need to stabilize quickly
> - Enis: What about 0.95.
> - Stack: Just do the snapshots thing. Every week, give a snapshot
> - Enis: we've done a bunch of stuff in RPC.. If we have to break
> something, we can if it is beta (0.96-beta).
> - Agreement is there generally ... Debate on the name with snapshot
> versus 95.0/96.0
>   -- 0.95 experimental .. 0.96 will be stable
>   -- If we go with 0.95, releasing will be easier as well..
>   -- 0.95 will not be in production..
>   -- 0.96 will be off 0.95 branch and not trunk based
> - How do we go about committing issues.. (issue commit rate is low)
>  -- 0.94 is stable -  2 new commits and 2 new bugs a day
>  -- 0.96 has lots of issues not reviewed
>  -- Break up the patch into multiple smaller pieces to make review easier
>  -- Branching on a big feature was suggested -
>    --- Issues: committer needs to be there
>          Sergey: If the rate of change is high on common code (to the
> branches) then merging will be tough
>    --- Jon: Refactor should be done in the main branch (since it
> doesn't add any new funtionality)
>    --- Release often to reduce #backports overall and issues with that..
> - Review process .. how to drive a review to closure. Effort goes
> waste if the review process is not completed. The same reviewer should
> continue to review the patch ..
> - Hard to enforce any process
> - Enis: there should be a summary of the patch and all that .. so that
> the review process is helped.. Hard to understand the architecture of
> the patch unless documented
> - Jon: It should be easy to make a one-to-one correspondence between
> the description and the patch
> - The commit should have only the jira# as opposed to pages of description
> - Component owners: is this working. Committers need to be forthcoming
> with reviews
>   -- Maybe review the modules and add some more if needed.
>   -- Good that we have more contributions coming than we have
> reviewers, but unless we keep up, we will plateau
>   -- Mail on dev@ list if review doesn't happen
> - Dev co-ordination:
>  -- How best can we pull together
>  -- Priorities:
>    --- Getting 0.96 out is priority
>    --- Backports to 0.94 will happen .. until 0.96 is stable
>    --- 0.92 release ? Any committer who wants to make a release can do
> so (maybe with some backports, etc.)
>    --- Backporting can be tough if there are bugs and the bugfix has
> to be applied to all branches
> - HBASE-2600 - this requires a change in the client and the server.
> They have to be changed in lock-step. Its hard to do this .. Jon
> doesn't want to have the fix for 0.96. So 0.98 might be another
> singular release. Maybe do a rewrite of the meta after taking a lock
> on the meta, do a shim layer to handle the backward compat. between
> 0.96 and 0.98
> - What do people want to get into 0.94
>  -- The biggest thing - Snapshots, mostly new code, about a 3rd of the
> stuff in 0.94 already
>  -- Compactions improvments - no backport
> - How devs can better co-ordinate
>   -- Snapshots co-ordination working well
>   -- One page design is useful (makes it readable and all)
>   -- How about handling the stripe compaction - where an idea leads to
> a bunch of others
>   -- Again write-up should be done
> - Should we change the description to match the comments
>   -- Two ideas suggested:
>    ---  We probably should have the description updated with the
> "Date: new description" if the issue at hand is updated
>    --- Should we have a summary after a bunch of comments - yes
> - The face-to-face meetings are useful. We should semd out the minutes
> of the discussion to the dev list. We probably should have more
> focused huddles. Discuss but don't decide (decide on the jira)!
> - Jon: Would people be amenable to merge sooner rather than later on
> snapshots? Tested and being beaten up.
> - Stack: Yes
> - What else goes in in 0.96:
>   -- RPC refactor
>   -- ROOT removal
>   -- Compaction stuff
>   -- Package name mailing list thread - there is now a jira on that.
> We shouldn't break clients. Package name changes is not worth the
> trouble.
> - A bunch of discussions on the RPC with KeyValue/Cells
> - What do we do about usability
>   -- It'll be nice if we don't need to change configs..
>   -- Maybe expose more metrics and then allow for online config
> changes since automatic config is difficult and needs to be battle
> tested and all.
> - Benchmarking of the release:
>   -- We should measure the overhead of PB stuff

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message