hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: HADOOP-7106 (project unsplit) this weekend
Date Mon, 13 Jun 2011 14:57:29 GMT
Hmm, as I got farther down my email this morning, I saw some complaints that
HBase's SVN was giving 403s as well, this morning. Perhaps there is some
ASF-wide issue going on, completely unrelated to the Hadoop changes over the
weekend?

-Todd

On Mon, Jun 13, 2011 at 7:52 AM, Todd Lipcon <todd@cloudera.com> wrote:

> Hey Robert,
>
> It seems to be working for me... what URL are you trying to check out?
> I moved aside my ~/.subversion dir, and then did:
> $ svn co http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/
>
> Thanks
> -Todd
>
> On Mon, Jun 13, 2011 at 7:05 AM, Robert Evans <evans@yahoo-inc.com> wrote:
>
>> Could someone unlock some of these branches for anonymous read only
>> checkout? At least with MR-279 I get a 403 forbidden error when I try to
>> check out.
>>
>> --Bobby
>>
>> On 6/12/11 6:38 PM, "Todd Lipcon" <todd@cloudera.com> wrote:
>>
>> OK, this seems to have succeeded without any big problems!
>>
>> I've re-enabled the git mirrors and the hudson builds. Feel free to commit
>> to the new trees.
>>
>> Here are some instructions for the migration:
>>
>> === SVN users ===
>>
>> Next time you "svn up" in your "common" working directory you'll end up
>> seeing the combined tree - ie a mapreduce/, hdfs/, and common/
>> subdirectory.
>> This is probably the easiest place from which to work, now. The URLs for
>> the
>> combined SVN trees are:
>>
>> trunk: https://svn.apache.org/repos/asf/hadoop/common/trunk/
>> branch-0.22:
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22
>> branch-0.21<http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22branch-0.21>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
>> yahoo-merge<http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21yahoo-merge>
>> :
>> http://svn.apache.org/repos/asf/hadoop/common/branches/yahoo-merge
>>  (this one has the yahoo-merge branches from common, hdfs, and mapred)
>> MR-279: http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>>  (this one has the yahoo-merge common and hdfs, and the MR-279 mapred)
>>
>> The same kind of thing happened for HDFS-1073 and branch-0.21-old.
>> Pre-project-split branches like branch-0.20 should have remained
>> untouched.
>>
>> You can proceed to delete your checkouts of the individual mapred and hdfs
>> trees, since they exist within the combined trees above. If for some
>> reason
>> you prefer to 'svn switch' an old MR or HDFS-specific checkout to point to
>> its new location, you can use the following incantation:
>> svn sw $(svn info | grep URL | awk '{print $2}' | sed
>> 's,\(hdfs\|mapreduce\|common\)/\(.*\),common/\2/\1,')
>>
>> === Git Users ===
>> The git mirrors of the above 7 branches should now have a set of 4 commits
>> near the top that look like this:
>>
>> Merge: 928d485 cd66945 77f628f
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:28 2011 +0000
>>
>>    HADOOP-7106. Reorganize SVN layout to combine HDFS, Common, and MR in a
>> single tree (project unsplit)
>>
>>    git-svn-id:
>> https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-<https://svn.apache.org/repos/asf/hadoop/common/trunk@113499413f79535-47bb-0310-9956-ffa450edef68>
>> 0310-9956-ffa450edef68
>>
>> commit 77f628ff5925c25ba2ee4ce14590789eb2e7b85b
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:27 2011 +0000
>>
>>    Relocate mapreduce into mapreduce/
>>
>> commit cd66945f62635f589ff93468e94c0039684a8b6d
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:26 2011 +0000
>>
>>    Relocate hdfs into hdfs/
>>
>> commit 928d485e2743115fe37f9d123ce9a635c5afb91a
>> Author: Todd Lipcon <todd@apache.org>
>> Date:   Sun Jun 12 22:53:25 2011 +0000
>>
>>    Relocate common into common/
>>
>> The first of these 4 is a 3-parent "octopus" merge commit of the
>> pre-project-unsplit branches. In theory, git is smart enough to track
>> changes through this merge, so long as you pass the right flags (eg
>> --follow). For example:
>>
>> todd@todd-w510:~/git/hadoop-common$ git log --pretty=oneline
>> --abbrev-commit
>> --follow mapreduce/src/java/org/apache/hadoop/mapred/JobTracker.java |
>> head
>> -10
>> 77f628f Relocate mapreduce into mapreduce/
>> 90df0cb MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
>> JobTrackerStatus.
>> ca2aba0 MAPREDUCE-2490. Add logging to graylist and blacklist activity to
>> aid diagnosis of related issues. Contributed by Jonathan Eagles
>> 32aaa2a MAPREDUCE-2515. MapReduce code references some deprecated options.
>> Contributed by Ari Rabkin.
>>
>> If you want to be able to have git follow renames all the way through the
>> project split back to the beginning of time, put the following in
>> hadoop-common/.git/info/grafts:
>> 5128a9a453d64bfe1ed978cf9ffed27985eeef36
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 6a3ac690e493c7da45bbf2ae2054768c427fd0e1
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>> 546d96754ffee3142bcbbf4563c624c053d0ed0d
>> 6c16dc8cf2b28818c852e95302920a278d07ad0c
>>
>> In terms of rebasing git branches, git is actually pretty smart. For
>> example, I have a local "HDFS-1073" branch in my hdfs repo. To transition
>> it
>> to the new combined repo, I did the following:
>>
>> # Add my project-split hdfs git repo as a remote:
>> git remote add splithdfs /home/todd/git/hadoop-hdfs/
>> git fetch splithdfs
>>
>> # Checkout a branch in my combined repo
>> git checkout -b HDFS-1073 splithdfs/HDFS-1073
>>
>> # Rebase it on the combined 1073 branch
>> git rebase origin/HDFS-1073
>>
>> ...and it actually applies my patches inside the appropriate subdirectory
>> (I
>> was surprised and impressed by this!)
>> If the branch you're rebasing has added or moved files, it might not be
>> smart enough and you'll have to manually rename them in your branch inside
>> of the appropriate subtree.. but for simple patches this seems to work.
>> For
>> less simple things, the best bet may be to use "git filter-branch" on the
>> patch series to relocate it inside a subdirectory, and then try to rebase.
>> Let me know if you need a hand with any git cleanup, happy to help.
>>
>>
>> == Outstanding issues ==
>>
>> The one outstanding issue I'm aware of is that the test-patch builds
>> should
>> be smart enough to be able to deal with patches that are relative to the
>> combined root instead of the original project. Right now, if you export a
>> diff from git, it will include "hdfs/" or "mapreduce/" in the changed file
>> names, and the QA bot won't know how to apply it. The workaround for this
>> is
>> to change directory into the relative subproject dir, and then pass
>> "--relative" to "git diff" or "git show", for example:
>>
>> todd@todd-w510:~/git/hadoop-common/mapreduce$ git diff --relative
>> --no-prefix
>> diff --git CHANGES.txt CHANGES.txt
>> ...
>>
>>
>> I imagine there are probably some other things that fell through the
>> cracks.
>> Please get in touch if there's anything that seems amiss.
>>
>> -Todd
>>
>>
>> On Sun, Jun 12, 2011 at 2:50 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>> > All of the nits I ran into should be resolved and we should be good to
>> go.
>> > I will start this in just about 10 minutes (3pm PST).
>> >
>> > ***Please hold all commits until further notice!*** I anticipate that
>> this
>> > should take under an hour, but if there are any bumps along the way it
>> might
>> > stretch into the evening. I'll send out an "all clear" email when things
>> are
>> > ready to go on the new layout.
>> >
>> > I've disabled all of the Hudson builds for now and will be re-enabling
>> them
>> > one by one after reconfiguring their SVN URLs.
>> >
>> > -Todd
>> >
>> > On Sat, Jun 11, 2011 at 8:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> >
>> >> Hi all,
>> >>
>> >> I'm figuring out one more small nit I noticed in my testing this
>> evening.
>> >> Hopefully I will figure out what's going wrong and be ready to press
>> the big
>> >> button tomorrow.
>> >>
>> >> Assuming I don't have to "abort mission", my hope is to do this at
>> around
>> >> 3PM PST tomorrow (Sunday). I'll send out a message asking folks to
>> please
>> >> hold commits to all branches while the move is in progress.
>> >>
>> >> Thanks
>> >> -Todd
>> >>
>> >>
>> >> On Fri, Jun 10, 2011 at 11:20 AM, Todd Lipcon <todd@cloudera.com>
>> wrote:
>> >>
>> >>> Hi all,
>> >>>
>> >>> Pending any unforeseen issues, I am planning on committing HADOOP-7106
>> >>> this weekend. I have the credentials from Jukka to take care of the
>> git
>> >>> trees as well, and have done a "practice" move several times on a
>> local
>> >>> mirror of the svn.
>> >>>
>> >>> I'll send out an announcement of the exact time in advance of when I
>> >>> actually do the commit.
>> >>>
>> >>> Thanks
>> >>> -Todd
>> >>> --
>> >>> Todd Lipcon
>> >>> Software Engineer, Cloudera
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Todd Lipcon
>> >> Software Engineer, Cloudera
>> >>
>> >
>> >
>> >
>> > --
>> > Todd Lipcon
>> > Software Engineer, Cloudera
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message