Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
From: Jeffrey Naisbitt <jnaisbit@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Mon, 13 Jun 2011 08:05:32 -0700
Subject: Re: HADOOP-7106 (project unsplit) this weekend
Thread-Topic: HADOOP-7106 (project unsplit) this weekend
Thread-Index: AcwpWg7IMsdIjZLTQm2/+zQZPXnG9AAgUgkz
Message-ID: <CA1B916C.78CF%jnaisbit@yahoo-inc.com>
In-Reply-To: <BANLkTikU1LxFc6aF_dV6Fbu4E-fxf3c7ww@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-Entourage/13.9.0.110114
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

When I checkout the yahoo-merge branch, I see these svn externals warnings:
svn: warning: Error handling externals definition for
'yahoo-merge/hdfs/src/test/bin':
svn: warning: URL=20
'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
revision 1135120 doesn't exist
svn: warning: Error handling externals definition for
'yahoo-merge/mapreduce/src/test/bin':
svn: warning: URL=20
'https://svn.apache.org/repos/asf/hadoop/common/trunk/src/test/bin' at
revision 1135120 doesn't exist

Also, the ant eclipse targets seem to be broken now.  It seems like various
parts of the eclipse target need to be commonized now (the
.eclipse-templates stuff and .classpath, .launches, etc.)

-Jeff


On 6/12/11 6:38 PM, "Todd Lipcon" <todd@cloudera.com> wrote:

> OK, this seems to have succeeded without any big problems!
>=20
> I've re-enabled the git mirrors and the hudson builds. Feel free to commi=
t
> to the new trees.
>=20
> Here are some instructions for the migration:
>=20
> =3D=3D=3D SVN users =3D=3D=3D
>=20
> Next time you "svn up" in your "common" working directory you'll end up
> seeing the combined tree - ie a mapreduce/, hdfs/, and common/ subdirecto=
ry.
> This is probably the easiest place from which to work, now. The URLs for =
the
> combined SVN trees are:
>=20
> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
> branch-0.22:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
> branch-0.21:
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
> yahoo-merge:
> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>   (this one has the yahoo-merge branches from common, hdfs, and mapred)
> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>   (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>=20
> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
> Pre-project-split branches like branch-0.20 should have remained untouche=
d.
>=20
> You can proceed to delete your checkouts of the individual mapred and hdf=
s
> trees, since they exist within the combined trees above. If for some reas=
on
> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point t=
o
> its new location, you can use the following incantation:
> svn sw $(svn info | grep URL | awk '{print $2}' | sed
> 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
>=20
> =3D=3D=3D Git Users =3D=3D=3D
> The git mirrors of the above 7 branches should now have a set of 4 commit=
s
> near the top that look like this:
>=20
> Merge: 928d485 cd66945 77f628f
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:28 2011 +0000
>=20
>     HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in=
 a
> single tree (project unsplit)
>=20
>     git-svn-id:
> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb=
-0310
> -9956-ffa450edef68
>=20
> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:27 2011 +0000
>=20
>     Relocate mapreduce into mapreduce/
>=20
> commit cd66945f62635f589ff93468e94c0039684a8b6d
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:26 2011 +0000
>=20
>     Relocate hdfs into hdfs/
>=20
> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
> Author: Todd Lipcon <todd@apache.org>
> Date:   Sun Jun 12 22:53:25 2011 +0000
>=20
>     Relocate common into common/
>=20
> The first of these 4 is a 3-parent "octopus" merge commit of the
> pre-project-unsplit branches. In theory, git is smart enough to track
> changes through this merge, so long as you pass the right flags (eg
> --follow). For example:
>=20
> todd@todd-w510:~/git/hadoop-common$ git log --pretty=3Doneline --abbrev-c=
ommit
> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java | he=
ad
> -10
> 77f628f Relocate mapreduce into mapreduce/
> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
> JobTrackerStatus.
> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
> aid diagnosis of related issues. Contributed by Jonathan Eagles
> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options=
.
> Contributed by Ari Rabkin.
>=20
> If you want to be able to have git follow renames all the way through the
> project split back to the beginning of time, put the following in
> hadoop-common/.git/info/grafts:
> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
> 546d96754ffee3142bcbbf4563c624c053d0ed0d
> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>=20
> In terms of rebasing git branches, git is actually pretty smart. For
> example, I have a local "HDFS-1073" branch in my hdfs repo. To transition=
 it
> to the new combined repo, I did the following:
>=20
> # Add my project-split hdfs git repo as a remote:
> git remote add splithdfs /home/todd/git/hadoop-hdfs/
> git fetch splithdfs
>=20
> # Checkout a branch in my combined repo
> git checkout -b HDFS-1073 splithdfs/HDFS-1073
>=20
> # Rebase it on the combined 1073 branch
> git rebase origin/HDFS-1073
>=20
> ...and it actually applies my patches inside the appropriate subdirectory=
 (I
> was surprised and impressed by this!)
> If the branch you're rebasing has added or moved files, it might not be
> smart enough and you'll have to manually rename them in your branch insid=
e
> of the appropriate subtree.. but for simple patches this seems to work. F=
or
> less simple things, the best bet may be to use "git filter-branch" on the
> patch series to relocate it inside a subdirectory, and then try to rebase=
.
> Let me know if you need a hand with any git cleanup, happy to help.
>=20
>=20
> =3D=3D Outstanding issues =3D=3D
>=20
> The one outstanding issue I'm aware of is that the test-patch builds shou=
ld
> be smart enough to be able to deal with patches that are relative to the
> combined root instead of the original project. Right now, if you export a
> diff from git, it will include "hdfs/" or "mapreduce/" in the changed fil=
e
> names, and the QA bot won't know how to apply it. The workaround for this=
 is
> to change directory into the relative subproject dir, and then pass
> "--relative" to "git diff" or "git show", for example:
>=20
> todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
> --no-prefix
> diff --git CHANGES.txt CHANGES.txt
> ...
>=20
>=20
> I imagine there are probably some other things that fell through the crac=
ks.
> Please get in touch if there's anything that seems amiss.
>=20
> -Todd
>=20
>=20
> On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>=20
>> All of the nits I ran into should be resolved and we should be good to g=
o.
>> I will start this in just about 10 minutes (3pm PST).
>>=20
>> ***Please hold all commits until further notice!*** I anticipate that th=
is
>> should take under an hour, but if there are any bumps along the way it m=
ight
>> stretch into the evening. I'll send out an "all clear" email when things=
 are
>> ready to go on the new layout.
>>=20
>> I've disabled all of the Hudson builds for now and will be re-enabling t=
hem
>> one by one after reconfiguring their SVN URLs.
>>=20
>> -Todd
>>=20
>> On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>=20
>>> Hi all,
>>>=20
>>> I'm figuring out one more small nit I noticed in my testing this evenin=
g.
>>> Hopefully I will figure out what's going wrong and be ready to press th=
e big
>>> button tomorrow.
>>>=20
>>> Assuming I don't have to "abort mission", my hope is to do this at arou=
nd
>>> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to plea=
se
>>> hold commits to all branches while the move is in progress.
>>>=20
>>> Thanks
>>> -Todd
>>>=20
>>>=20
>>> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com> wrote=
:
>>>=20
>>>> Hi all,
>>>>=20
>>>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>>>> this weekend. I have the credentials from Jukka to take care of the gi=
t
>>>> trees as well, and have done a "practice" move several times on a loca=
l
>>>> mirror of the svn.
>>>>=20
>>>> I'll send out an announcement of the exact time in advance of when I
>>>> actually do the commit.
>>>>=20
>>>> Thanks
>>>> -Todd
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>=20
>>>=20
>>>=20
>>>=20
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>=20
>>=20
>>=20
>>=20
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>=20
>=20
>=20