hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gray <jg...@facebook.com>
Subject RE: Merging 0.20 BRANCH and TRUNK WAS -> Re: JIRAs committed on trunk but not branch (take 2)
Date Wed, 12 May 2010 15:16:12 GMT
Thanks for digging more stack.

I agree that after going over the diffs between branch and trunk, the better choice is to
switch to trunk and merge in the stuff that went into branch that didn't make it into trunk.
 I went through the list of what's in trunk and not branch twice last night and there's really
nothing significant outside of replication, test refactoring, shell stuff, and some HLog fixes/refactors.
 Lots of stability/testing work still needs to be done around HLog and recovery but we should
do it against trunk where some good stuff has already gone in.

However, I'm only for doing this if we can get it done very soon.  It's going to be disruptive
to those of us working against branch right now and so the sooner the better.  It doesn't
make sense to start developing on trunk before we merge stuff from branch in, especially HBASE-2248.
 Of the list you made Stack, it seems like this is the only critical piece that also significantly
changes large portions of the code.

Some of the other smaller jiras that were done against branch only could trickle in after
the move, if necessary, but 2248 at least appears to be a blocker to doing this transition.

Ryan, are you comfortable with this?

If Ryan can do the port of 2248 to trunk this week, then I'm +1 on merging branch into trunk.
 I would then think that in a 2-3 week timeframe we would cut a new branch off of trunk and
stabilize it.

I would vote for making that branch 0.90.  I would also vote for doing anything else big/disruptive/compatibility
breaking at the same time we all move onto trunk and kill the branch.  For example, the switch
to org.apache.hbase should happen now rather than later.  Since trunk also includes some versioning
stuff in the RPC, we'll be able to add stuff to the API moving forward without breaking client
compatibility, so it seems like breaking it all now is a good idea.  We may not need to break
it again for whatever comes after 0.90.  Also remember, the old APIs are already ripped out
of trunk.

Anyone with major objections to any of this?  We should move to a vote soon.

Also, I think we should adopt the practice of having a Release Manager for when we cut the
branch off trunk.  I'm happy to do it but that doesn't need to be determined now.  This is
widely used around other Apache projects and just now being adopted by Hadoop.  If you aren't
familiar with how this works in other projects, check out how they do it in httpd (http://httpd.apache.org/dev/release.html)
or the discussion on the hadoop list (http://mail-archives.apache.org/mod_mbox/hadoop-general/201005.mbox/%3Ch2q1267dd3b1005041331r7d8f696di370a279ff605832f@mail.gmail.com%3E).

JG

> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of
> Stack
> Sent: Tuesday, May 11, 2010 10:45 PM
> To: hbase-dev@hadoop.apache.org
> Subject: Merging 0.20 BRANCH and TRUNK WAS -> Re: JIRAs committed on
> trunk but not branch (take 2)
>
> On IRC we were going back and forth about what to do about our current
> state where developer energy is all up on the 0.20 branch with TRUNK
> near neglected with all driving at the next release, the one that will
> exploit hdfs-200, -142, et al. to get us data durability.  We were
> debating whether we should merge TRUNK into 0.20 (because 0.20 has
> most dev energy at moment and I think consensus is that it more
> 'stable' than the less tested TRUNK) or rather, that we merge 0.20 to
> TRUNK.
>
> After doing a compare of TRUNK and 0.20 BRANCH CHANGELOGs, it seems
> pretty obvious that we should be merging the 0.20 branch to TRUNK.
> The amount of items that are in the branch and not in TRUNK are few
> and relatively minor with the exception of HBASE-2248.  The changes in
> TRUNK that are not on the branch look like fixes that can only benefit
> -- mostly small fixes and improvements to things like our mapreduce
> hookups and shell with little to no destabilizing changes other than
> the move to maven and the replication changes.
>
> Here are the changes that are on the 0.20 branch but not in TRUNK:
> http://gist.github.com/398246 (A good few of these can't be ported to
> TRUNK anyways because, for instance, they are fixes for facility that
> does not exist in TRUNK).
>
> Here is what is in TRUNK that is not on the branch:
> http://gist.github.com/398253
>
> St.Ack
>
>
> On Tue, May 11, 2010 at 7:21 PM, Jonathan Gray <jgray@facebook.com>
> wrote:
> > Where 0.21 is the new 0.20.5, perhaps to be released as 0.90? :)
> >
> >> -----Original Message-----
> >> From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> >> Sent: Tuesday, May 11, 2010 7:18 PM
> >> To: hbase-dev@hadoop.apache.org
> >> Subject: Re: JIRAs committed on trunk but not branch (take 2)
> >>
> >> I think 0.21 is the time to move to o.a.hbase.  Pushing it out makes
> >> it more painful, and it's just a package name change at least.
> >>
> >> -ryan
> >>
> >> On Tue, May 11, 2010 at 7:14 PM, Jonathan Gray <jgray@facebook.com>
> >> wrote:
> >> > Also, trunk has the deprecated APIs removed.  I'm +1 on that for
> our
> >> next release.
> >> >
> >> > What are peoples thoughts on moving to o.a.h?  Personally this
> looks
> >> like a good opportunity to do it as we're all going to need to
> rebase
> >> existing work onto trunk anyways, we're breaking compatibility
> across
> >> the board, etc.  It would be nice not to have to break compatibility
> >> for the release that follows our next one.
> >> >
> >> > JG
> >> >
> >> >> -----Original Message-----
> >> >> From: Jonathan Gray [mailto:jgray@facebook.com]
> >> >> Sent: Tuesday, May 11, 2010 7:12 PM
> >> >> To: hbase-dev@hadoop.apache.org
> >> >> Subject: RE: JIRAs committed on trunk but not branch (take 2)
> >> >>
> >> >> Agreed.  I went through it and there doesn't seem to be anything
> >> major
> >> >> besides replication and HLog stuff.
> >> >>
> >> >> We should probably go through all the commits on branch to make
> sure
> >> it
> >> >> all got into trunk as well.
> >> >>
> >> >> > -----Original Message-----
> >> >> > From: Ryan Rawson [mailto:ryanobjc@gmail.com]
> >> >> > Sent: Tuesday, May 11, 2010 6:59 PM
> >> >> > To: hbase-dev@hadoop.apache.org
> >> >> > Subject: Re: JIRAs committed on trunk but not branch (take 2)
> >> >> >
> >> >> > This is a big list, but not many things are significant... Lots
> of
> >> >> > little fixes, fixes due to first ivy then maven. Other things
> we
> >> >> > should have (such as junit4 and other speed improvements)
> anyways,
> >> >> > etc.
> >> >> >
> >> >> >
> >> >> > On Tue, May 11, 2010 at 5:40 PM, Jonathan Gray
> >> <jgray@facebook.com>
> >> >> > wrote:
> >> >> > > Looked botched to me on receipt... trying again:
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > Below is a list I compiled of trunk-only JIRAs.
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > This list contains JIRAs that are Fixed and have a Fix
> Version
> >> of
> >> >> > 0.21 and no other versions (so only as accurate as the jira fix
> >> >> > versions are).
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > [HBASE-410] [testing] Speed up the test suite
> >> >> > >
> >> >> > > [HBASE-1276] [testing] Upgrade to JUnit 4.x and use
> @BeforeClass
> >> >> > annotations to optimize tests
> >> >> > >
> >> >> > > [HBASE-1360] move up to Thrift 0.2.0
> >> >> > >
> >> >> > > [HBASE-1373] Update Thrift to use compact/framed protocol
> >> >> > >
> >> >> > > [HBASE-1433] Update hbase build to match core, use ivy,
> publish
> >> >> jars
> >> >> > to maven repo, etc.
> >> >> > >
> >> >> > > [HBASE-1444] Use new HADOOP-4829 shutdown flag
> >> >> > >
> >> >> > > [HBASE-1537] Intra-row scanning
> >> >> > >
> >> >> > > [HBASE-1614] single zk node buckling under small node?
> >>  connections
> >> >> > never timing out?
> >> >> > >
> >> >> > > [HBASE-1642] Add main to HRegion that can read a passed
> .META.
> >> >> > >
> >> >> > > [HBASE-1643] ScanDeleteTracker takes comparator but it unused
> >> >> > >
> >> >> > > [HBASE-1728] Column family scoping and cluster identification
> >> >> > >
> >> >> > > [HBASE-1756] Refactor HLog
> >> >> > >
> >> >> > > [HBASE-1758] Extract interface out of HTable
> >> >> > >
> >> >> > > [HBASE-1776] Make rowcounter enum public
> >> >> > >
> >> >> > > [HBASE-1778] Improve PerformanceEvaluation
> >> >> > >
> >> >> > > [HBASE-1820] Update jruby from 1.2 to 1.3.1
> >> >> > >
> >> >> > > [HBASE-1822] Remove the deprecated APIs
> >> >> > >
> >> >> > > [HBASE-1825] code cleanup, hmaster split debug logs
> >> >> > >
> >> >> > > [HBASE-1835] Add more delete tests
> >> >> > >
> >> >> > > [HBASE-1841] If multiple of same key in an hfile and they
> span
> >> >> > blocks, may miss the earlier keys on a lookup
> >> >> > >
> >> >> > > [HBASE-1848] Fixup shell for HBASE-1822
> >> >> > >
> >> >> > > [HBASE-1850] src/examples/mapred do not compile after HBASE-
> 1822
> >> >> > >
> >> >> > > [HBASE-1869] IndexedTable delete fails when used in
> conjunction
> >> >> with
> >> >> > RowLock()
> >> >> > >
> >> >> > > [HBASE-1885] Simplify use of IndexedTable outside Java API
> >> >> > >
> >> >> > > [HBASE-1887] Update hbase trunk to latests on hadoop 0.21
> branch
> >> so
> >> >> > we can all test sync/append
> >> >> > >
> >> >> > > [HBASE-1889] ClassNotFoundException on trunk for REST
> >> >> > >
> >> >> > > [HBASE-1893] hdfs-127 and native libs for hbase 0.21
> >> >> > >
> >> >> > > [HBASE-1901] "General" partitioner for "hbase-48" bulk
> (behind
> >> the
> >> >> > api, write hfiles direct) uploader
> >> >> > >
> >> >> > > [HBASE-1902] Let PerformanceEvaluation support setting
> tableName
> >> >> and
> >> >> > compress algorithm
> >> >> > >
> >> >> > > [HBASE-1907] Version all client writables
> >> >> > >
> >> >> > > [HBASE-1914] hlog should be able to set replication level
for
> >> the
> >> >> log
> >> >> > indendently from any other files
> >> >> > >
> >> >> > > [HBASE-1915] HLog.sync is called way too often, needs to
be
> only
> >> >> > called 1x per RPC
> >> >> > >
> >> >> > > [HBASE-1926] Remove unused xmlenc jar from trunk
> >> >> > >
> >> >> > > [HBASE-1930] Put.setTimeStamp misleading (doesn't change
> >> timestamp
> >> >> on
> >> >> > existing KeyValues, not copied in copy constructor)
> >> >> > >
> >> >> > > [HBASE-1933] Upload Hbase jars to a public maven repository
> >> >> > >
> >> >> > > [HBASE-1939] HLog group commit
> >> >> > >
> >> >> > > [HBASE-1942] Update hadoop jars in trunk; update to r831142
> >> >> > >
> >> >> > > [HBASE-1943] Remove AgileJSON; unused.
> >> >> > >
> >> >> > > [HBASE-1944] Add a "deferred log flush" attribute to HTD
> >> >> > >
> >> >> > > [HBASE-1945] Remove META and ROOT memcache size bandaid
> >> >> > >
> >> >> > > [HBASE-1963] Output to multiple tables from Hadoop MR without
> >> use
> >> >> of
> >> >> > HTable
> >> >> > >
> >> >> > > [HBASE-1971] Unit test the full WAL replay cycle
> >> >> > >
> >> >> > > [HBASE-1974] Update to latest on hadoop 0.21 branch
> >> (November11th,
> >> >> > 2009)
> >> >> > >
> >> >> > > [HBASE-1977] Add ts and allow setting VERSIONS when scanning
> in
> >> >> shell
> >> >> > >
> >> >> > > [HBASE-1995] Add configurable max value size check
> >> >> > >
> >> >> > > [HBASE-1996] Configure scanner buffer in bytes instead of
> number
> >> of
> >> >> > rows
> >> >> > >
> >> >> > > [HBASE-2013] Add useful helpers to HBaseTestingUtility.java
> >> >> > >
> >> >> > > [HBASE-2017] Set configurable max value size check to 10MB
> >> >> > >
> >> >> > > [HBASE-2028] Add HTable.incrementColumnValue() to shell
> >> >> > >
> >> >> > > [HBASE-2036] Use Configuration instead of HBaseConfiguration
> >> >> > >
> >> >> > > [HBASE-2040] Fixes to group commit
> >> >> > >
> >> >> > > [HBASE-2041] Change WAL default configuration values
> >> >> > >
> >> >> > > [HBASE-2044] HBASE-1822 removed not-deprecated APIs
> >> >> > >
> >> >> > > [HBASE-2053] Upper bound of outstanding WALs can be overrun
> >> >> > >
> >> >> > > [HBASE-2059] Break out WAL reader and writer impl from HLog
> >> >> > >
> >> >> > > [HBASE-2070] Collect HLogs and delete them after a period
of
> >> time
> >> >> > >
> >> >> > > [HBASE-2072] fs.automatic.close isn't passed to FileSystem
> >> >> > >
> >> >> > > [HBASE-2085] StringBuffer -&gt; StringBuilder - conversion
of
> >> >> > references as necessary
> >> >> > >
> >> >> > > [HBASE-2086] Job(configuration,String) deprecated
> >> >> > >
> >> >> > > [HBASE-2089] HBaseConfiguration() ctor. deprecated
> >> >> > >
> >> >> > > [HBASE-2090] findbugs issues
> >> >> > >
> >> >> > > [HBASE-2107] Upgrading Lucene 2.2 to Lucene 3.0.0
> >> >> > >
> >> >> > > [HBASE-2114] Can't start HBase in trunk
> >> >> > >
> >> >> > > [HBASE-2128] ant tar build broken since switch to Ivy
> >> >> > >
> >> >> > > [HBASE-2130] bin/* scripts - not to include lib/test/**/*.jar
> >> >> > >
> >> >> > > [HBASE-2134] Ivy nit regarding checking with latest snapshots
> >> >> > >
> >> >> > > [HBASE-2135] ant javadoc complains about missing classes
> >> >> > >
> >> >> > > [HBASE-2136] Forward-port the old mapred package
> >> >> > >
> >> >> > > [HBASE-2137] javadoc warnings from 'javadoc' target
> >> >> > >
> >> >> > > [HBASE-2139] findbugs task in build.xml
> >> >> > >
> >> >> > > [HBASE-2140] findbugs issues - 2 performance warnings as
> >> suggested
> >> >> by
> >> >> > findbugs
> >> >> > >
> >> >> > > [HBASE-2150] Deprecated HBC(Configuration) constructor
> doesn't
> >> call
> >> >> > this()
> >> >> > >
> >> >> > > [HBASE-2151] Remove onelab and include generated thrift
> classes
> >> in
> >> >> > javadoc
> >> >> > >
> >> >> > > [HBASE-2153] Publish generated HTML documentation for Thrift
> on
> >> the
> >> >> > website
> >> >> > >
> >> >> > > [HBASE-2163] ZK dependencies - explicitly add them until
ZK
> >> >> artifacts
> >> >> > are published to mvn repository
> >> >> > >
> >> >> > > [HBASE-2164] Ivy nit - clean up configs
> >> >> > >
> >> >> > > [HBASE-2172] Add constructor to Put for row key and timestamp
> >> >> > >
> >> >> > > [HBASE-2178] Hooks for replication
> >> >> > >
> >> >> > > [HBASE-2184] Calling HTable.getTableDescriptor().* on a full
> >> >> cluster
> >> >> > takes a long time
> >> >> > >
> >> >> > > [HBASE-2194] HTable - put(Put) , put(List&lt;Put) code
> >> duplication
> >> >> > >
> >> >> > > [HBASE-2209] Support of List [ ] in HBaseOutputWritable for
> >> >> > serialization
> >> >> > >
> >> >> > > [HBASE-2211] Add a new Filter that checks a single column
> value
> >> but
> >> >> > does not emit it.
> >> >> > >
> >> >> > > [HBASE-2212] Refactor out lucene dependencies from HBase
> >> >> > >
> >> >> > > [HBASE-2221] MR to copy a table
> >> >> > >
> >> >> > > [HBASE-2224] Broken build:
> >> >> > TestGetRowVersions.testGetRowMultipleVersions
> >> >> > >
> >> >> > > [HBASE-2226] HQuorumPeerTest doesnt run because it doesnt
> start
> >> >> with
> >> >> > the word Test
> >> >> > >
> >> >> > > [HBASE-2245] Unnecessary call to syncWal(region); in
> >> HRegionServer
> >> >> > >
> >> >> > > [HBASE-2246] Add a getConfiguration method to HTableInterface
> >> >> > >
> >> >> > > [HBASE-2254] Improvements to the Maven POMs
> >> >> > >
> >> >> > > [HBASE-2255] take trunk back to hadoop 0.20
> >> >> > >
> >> >> > > [HBASE-2260] Remove all traces of Ant and Ivy
> >> >> > >
> >> >> > > [HBASE-2264] Adjust the contrib apps to the Maven project
> layout
> >> >> > >
> >> >> > > [HBASE-2266] [stargate] missing MiniDFSCluster dependency
> >> >> > >
> >> >> > > [HBASE-2267] More improvements to the Maven build
> >> >> > >
> >> >> > > [HBASE-2268] [stargate] "Failed tests:
> >> >> > warning(junit.framework.TestSuite$1)" and DEBUG output is
> dumped
> >> to
> >> >> > console since move to Mavenized build
> >> >> > >
> >> >> > > [HBASE-2276] Hbase Shell hcd() method is broken by the
> >> replication
> >> >> > scope parameter
> >> >> > >
> >> >> > > [HBASE-2279] Hbase Shell does not have any tests
> >> >> > >
> >> >> > > [HBASE-2281] Hbase shell does not work when started from
the
> >> build
> >> >> > dir
> >> >> > >
> >> >> > > [HBASE-2282] More directories should be ignored when using
> git
> >> for
> >> >> > development
> >> >> > >
> >> >> > > [HBASE-2309] Add apache releases to pom (list of )
> repositories
> >> >> > >
> >> >> > > [HBASE-2313] Nit-pick about hbase-2279 shell fixup, if you
do
> >> get
> >> >> > with non-existant column family, throws lots of exceptions
> >> >> > >
> >> >> > > [HBASE-2314] [shell] Support for getting counters
> >> >> > >
> >> >> > > [HBASE-2316] Need an ability to run shell tests w/o invoking
> >> junit
> >> >> > >
> >> >> > > [HBASE-2324] Refactoring of TableRecordReader (mapred /
> >> mapreduce)
> >> >> > for reuse outside the scope of InputSplit / RecordReader
> >> >> > >
> >> >> > > [HBASE-2331] [shell] count command needs a way to specify
> scan
> >> >> > caching
> >> >> > >
> >> >> > > [HBASE-2334] Slimming of Maven dependency tree - improves
> >> assembly
> >> >> > build speed,
> >> >> > >
> >> >> > > [HBASE-2336] Fix build broken with HBASE-2334
> >> >> > >
> >> >> > > [HBASE-2348] Stargate needs both JAR and WAR artifacts
> >> >> > >
> >> >> > > [HBASE-2361] WALEdit broke replication scope
> >> >> > >
> >> >> > > [HBASE-2364] Ignore Deprecations during build
> >> >> > >
> >> >> > > [HBASE-2370] saveVersion.sh doesnt properly grab the git
> >> revision
> >> >> > >
> >> >> > > [HBASE-2374] TableInputFormat - Configurable parameter to
add
> >> >> column
> >> >> > families
> >> >> > >
> >> >> > > [HBASE-2430] Disable frag display in trunk, let HBASE-2165
> >> replace
> >> >> it
> >> >> > >
> >> >> > > [HBASE-2435] HTablePool - method to release resources after
> use
> >> >> > >
> >> >> > > [HBASE-2452] Fix our Maven dependencies
> >> >> > >
> >> >> > > [HBASE-2463] Various Bytes.* functions silently ignore
> invalid
> >> >> > arguments
> >> >> > >
> >> >> > > [HBASE-2491] master.jsp uses absolute links to table.jsp.
> This
> >> >> broke
> >> >> > when master.jsp moved under webapps/master
> >> >> > >
> >> >> > > [HBASE-2494] Does not apply new.name parameter to CopyTable.
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > JG
> >> >> > >
> >> >> > >
> >> >> > >
> >> >
> >

Mime
View raw message