Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@asterixdb.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAN_YF5zkcn3b6Wn3OPkJOe7GocsTr-eDHVL=e4hRO1uaumhoEg@mail.gmail.com>
References: 
 <CAN_YF5w0=Krfj0-frVzrFT-SivDnWrfQf6d8U0DKKp-E5p9=cA@mail.gmail.com>
 <71F2D731-2B41-49FC-9F27-1A00BB58C87F@apache.org>
 <CAN_YF5yuwjZAL1uDAeYapLK2_vp942Xw1suy_98zK5ZVPhijbQ@mail.gmail.com>
 <CAN_YF5yYN-20X3z5-y_M+i7+rotR07i8aY3Z0HeB6_2sgxr7Zw@mail.gmail.com>
 <F9FE3AB6-6B84-44E0-A4F0-D3C49126B49B@apache.org>
 <CAN_YF5w9wSDjaaFqH7OuLDO_YJ+XQbSJqj1VQ-360LouHJwMVg@mail.gmail.com>
 <84DF74BD-8BE2-42C0-B449-D9A18A29DC78@apache.org>
 <CAE1xAntJ0myHC0jG-Y4ouayuEw=3L8Co3LPZWNtrXXKa0YyShg@mail.gmail.com>
 <CAE1xAnvA0=uRMo8f9OEUkcQx-7+pQHLqzY-hdJuqUTgNNHzdqg@mail.gmail.com>
 <CAE1xAnsDYX+b_5ZSgQm2Laq6_CrndtPCFAPZv9viYZE7W06etQ@mail.gmail.com>
 <5702CBDE.5090407@gmail.com>
 <CAE1xAnugE-5K=XUptRkwuHrFjgNt+xOW4YqnoZ=m2v+uk9hvNQ@mail.gmail.com>
 <CAE1xAnuGbVPtxDgJXsRngxqgtU7jxM-LmcRg9q3KG+Y-5bomqg@mail.gmail.com>
 <CAN_YF5yT54XT52DWcnqB43Q1_m8DJdtZQ1bttRG4o14+dHHTWw@mail.gmail.com>
 <CAN_YF5zkcn3b6Wn3OPkJOe7GocsTr-eDHVL=e4hRO1uaumhoEg@mail.gmail.com>
From: Ian Maxon <imaxon@uci.edu>
Date: Thu, 7 Apr 2016 09:12:40 -0700
Message-ID: 
 <CAN_YF5xrnGxzDLh-A7cDxOaokzksgO439cXh4sYm+KM6+BNq9g@mail.gmail.com>
Subject: Re: The Great Merge
To: dev@asterixdb.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a113d2d662a7cc7052fe75a67

--001a113d2d662a7cc7052fe75a67
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Also, please don't push any new changes to Hyracks. You will need to merge
these into your asterix change or a new change if it is a hyracks change
with no asterixdb content. I have disabled the jenkins job that verifies
hyracks patches on Gerrit so hopefully nothing gets through.

On Thu, Apr 7, 2016 at 8:02 AM, Ian Maxon <imaxon@uci.edu> wrote:

> Alright, everything is pushed. Please post here or start another thread i=
f
> you start to experience issues.
>
> On Tue, Apr 5, 2016 at 12:21 PM, Ian Maxon <imaxon@uci.edu> wrote:
>
>> Just as a reminder, I'll be merging this tommorow since it seems like
>> everything should go fairly smoothly.
>>
>> On Mon, Apr 4, 2016 at 1:20 PM, Steven Jacobs <sjaco002@ucr.edu> wrote:
>>
>>> http://landbeforetime.wikia.com/wiki/Great_Valley
>>>
>>> On Mon, Apr 4, 2016 at 1:19 PM, Steven Jacobs <sjaco002@ucr.edu> wrote:
>>>
>>> > No, we are living in the GREAT valley :)
>>> > Steven
>>> >
>>> > On Mon, Apr 4, 2016 at 1:17 PM, Mike Carey <dtabass@gmail.com> wrote:
>>> >
>>> >> Sounds like things are GOOD!  Excellent.  (So not to be feared like
>>> the
>>> >> event that the name of this one keeps reminding me of:
>>> >> http://landbeforetime.wikia.com/wiki/Great_Earthshake :-).)
>>> >>
>>> >>
>>> >> On 4/4/16 1:12 PM, Steven Jacobs wrote:
>>> >>
>>> >>> It seems that I might be the only one concerned here, but it seems
>>> like
>>> >>> there should be others, so I am continuing this thread.
>>> >>>
>>> >>> I modified the perl REGEX from Chris' summer solution, and it works=
!
>>> >>>
>>> >>> Once Ian has merged master:
>>> >>>
>>> >>> 1. On your local branch, find the *parent* of the first commit you
>>> want
>>> >>> to
>>> >>> migrate onto the new master, e.g.
>>> >>> de6e0da24c26037967eb9a937d2c77c6c43e8761
>>> >>>
>>> >>> 2. Run this magic command:
>>> >>>
>>> >>>     git format-patch --stdout
>>> de6e0da24c26037967eb9a937d2c77c6c43e8761 |
>>> >>> perl -pe 's#asterix-#asterixdb/asterix-#g' > /tmp/my.patch
>>> >>>
>>> >>> 3. Now fetch master, and create a new local branch from it:
>>> >>>
>>> >>>     git switch master; git pull; git checkout -B newbranch
>>> >>>
>>> >>> 4. Apply your tweaked patch:
>>> >>>
>>> >>>     git am /tmp/my.patch
>>> >>>
>>> >>>
>>> >>> This recognized ALL of my file moves/renames and applied them
>>> correctly.
>>> >>> It
>>> >>> leaves only two issues:
>>> >>> 1) Something similar will probably need to be done for Hyracks
>>> changes
>>> >>> 2) My pom changes didn't apply. This isn't so bad since there are
>>> only a
>>> >>> few pom files total.
>>> >>>
>>> >>>
>>> >>> I hope this helps,
>>> >>> Steven
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Fri, Apr 1, 2016 at 11:31 AM, Steven Jacobs <sjaco002@ucr.edu>
>>> wrote:
>>> >>>
>>> >>> Here is Chris's original solution to give context. I think changing
>>> the
>>> >>>> REGEX might be enough to re-use the solution:
>>> >>>>
>>> >>>> 1. On your local branch, find the *parent* of the first commit you
>>> want
>>> >>>> to
>>> >>>> migrate onto the new master. If you were fully up-to-date before t=
he
>>> >>>> repackaging commits went in, this will be Till's
>>> >>>> change 95350e253f3462b1fb8d08396b4fddadaa33bf53, so I'll use that
>>> here.
>>> >>>>
>>> >>>> 2. Run this magic command:
>>> >>>>
>>> >>>>     git format-patch --stdout
>>> 95350e253f3462b1fb8d08396b4fddadaa33bf53 |
>>> >>>> perl -pe 's#edu(.)uci.ics#org\1apache#g' > /tmp/my.patch
>>> >>>>
>>> >>>> 3. Now fetch the new master, and create a new local branch from it=
:
>>> >>>>
>>> >>>>     git switch master; git pull; git checkout -B newbranch
>>> >>>>
>>> >>>> 4. Apply your tweaked patch:
>>> >>>>
>>> >>>>     git am /tmp/my.patch
>>> >>>>
>>> >>>>
>>> >>>> Steven
>>> >>>>
>>> >>>> On Fri, Apr 1, 2016 at 11:07 AM, Steven Jacobs <sjaco002@ucr.edu>
>>> >>>> wrote:
>>> >>>>
>>> >>>> I've tried doing this now on my branch.
>>> >>>>> As I feared, all of the files that are renamed/moved become
>>> conflicts
>>> >>>>> (just a few hundred conflicts in my case =F0=9F=98=91).
>>> >>>>> I'm wondering if we could use a similar technique for what we did
>>> >>>>> during
>>> >>>>> the summer (for the apache change) to get around this.
>>> >>>>>
>>> >>>>> Steven
>>> >>>>>
>>> >>>>> On Fri, Apr 1, 2016 at 9:40 AM, Till Westmann <tillw@apache.org>
>>> >>>>> wrote:
>>> >>>>>
>>> >>>>> I=E2=80=99m not sure I completely understand what you are saying.=
 Is this a
>>> >>>>>> temporary state that will get cleaned up later or is this
>>> supposed to
>>> >>>>>> stay this way (having "-fullstack" in the names)?
>>> >>>>>>
>>> >>>>>> Thanks,
>>> >>>>>> Till
>>> >>>>>>
>>> >>>>>>
>>> >>>>>> On 31 Mar 2016, at 19:39, Ian Maxon wrote:
>>> >>>>>>
>>> >>>>>> I'm not sure if it was necessary to rename it, but the original
>>> issue
>>> >>>>>> is
>>> >>>>>>
>>> >>>>>>> that the hyracks repo itself has a folder named hyracks, that
>>> >>>>>>> contains
>>> >>>>>>> hyracks. I thought this might confuse git if I did something li=
ke
>>> >>>>>>> make a
>>> >>>>>>> new temporary folder, move everything into that, and then renam=
e
>>> it
>>> >>>>>>> to
>>> >>>>>>> 'hyracks'.
>>> >>>>>>>
>>> >>>>>>> On Thu, Mar 31, 2016 at 6:35 PM, Till Westmann <tillw@apache.or=
g
>>> >
>>> >>>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>> Interesting!
>>> >>>>>>>
>>> >>>>>>>> One thing I=E2=80=99m wondering about is why you=E2=80=99ve ad=
ded "-fullstack"
>>> to
>>> >>>>>>>> the
>>> >>>>>>>> artifactId and the hyracks module.
>>> >>>>>>>>
>>> >>>>>>>> Cheers,
>>> >>>>>>>> Till
>>> >>>>>>>>
>>> >>>>>>>>
>>> >>>>>>>> On 31 Mar 2016, at 17:21, Ian Maxon wrote:
>>> >>>>>>>>
>>> >>>>>>>> I've gone ahead and tried merging my topic branch with this
>>> change,
>>> >>>>>>>> and it
>>> >>>>>>>>
>>> >>>>>>>> turned out surprisingly well. I really didn't have many issues=
.
>>> I'll
>>> >>>>>>>>> summarize the process:
>>> >>>>>>>>>
>>> >>>>>>>>> 1) Merge the change from asterixdb with your topic branch
>>> checked
>>> >>>>>>>>> out, so
>>> >>>>>>>>> just 'git merge hyracks-merge2'.
>>> >>>>>>>>> The only real conflict should be the pom, if you altered that=
.
>>> I
>>> >>>>>>>>> found it
>>> >>>>>>>>> easiest to just replicate my changes and take the upstream,
>>> rather
>>> >>>>>>>>> than
>>> >>>>>>>>> trying anything funny, since usually pom changes are not majo=
r.
>>> >>>>>>>>>
>>> >>>>>>>>> 2) Add your hyracks folder as a remote (for me, 'git remote a=
dd
>>> >>>>>>>>> hyracks-local file:///home/...')
>>> >>>>>>>>>
>>> >>>>>>>>> 3) Merge your hyracks topic branch into asterixdb ( ' git mer=
ge
>>> >>>>>>>>> hyracks-local/imaxon/hdfs')
>>> >>>>>>>>> This also worked pretty well, the only extra hiccup besides
>>> the pom
>>> >>>>>>>>> was
>>> >>>>>>>>> files I had created. Those appeared at the top level again
>>> after
>>> >>>>>>>>> the
>>> >>>>>>>>> merge.
>>> >>>>>>>>> But, all you have to do is move them back down one folder int=
o
>>> >>>>>>>>> hyracks-fullstack.
>>> >>>>>>>>>
>>> >>>>>>>>> That's about it really. I went ahead and pushed this up to
>>> github
>>> >>>>>>>>> as
>>> >>>>>>>>> well
>>> >>>>>>>>> so if anyone would like to take a look at the process or chec=
k
>>> out
>>> >>>>>>>>> the
>>> >>>>>>>>> branch to see what happened (at least for me), the branch is
>>> here:
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> >>>>>>>>>
>>> https://github.com/parshimers/incubator-asterixdb/tree/imaxon/hdfs-plus=
-hyracks
>>> >>>>>>>>>
>>> >>>>>>>>> Thanks,
>>> >>>>>>>>> -Ian
>>> >>>>>>>>>
>>> >>>>>>>>> On Wed, Mar 30, 2016 at 6:17 PM, Ian Maxon <imaxon@uci.edu>
>>> wrote:
>>> >>>>>>>>>
>>> >>>>>>>>> Chris found an issue with the way git histories were being
>>> handled
>>> >>>>>>>>> in
>>> >>>>>>>>> the
>>> >>>>>>>>>
>>> >>>>>>>>> way I merged things, so I have revised the proposed branch:
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> https://github.com/parshimers/incubator-asterixdb/commits/hyracks-merge=
2
>>> >>>>>>>>>>
>>> >>>>>>>>>> Basically I was trying to fit everything into one commit,
>>> because
>>> >>>>>>>>>> I
>>> >>>>>>>>>> thought at first that I could submit it to Gerrit that way.
>>> >>>>>>>>>> However
>>> >>>>>>>>>> that
>>> >>>>>>>>>> doesn't work for other reasons, basically Gerrit tries to
>>> treat
>>> >>>>>>>>>> every new
>>> >>>>>>>>>> commit from Hyracks as a new change. Splitting the commits o=
f
>>> the
>>> >>>>>>>>>> repository merge fixes the issue.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> @Till, I think that creating a textual patch would just be
>>> more
>>> >>>>>>>>>> work. If
>>> >>>>>>>>>> I
>>> >>>>>>>>>> were to do it that way I would try fetching the Gerrit patch=
,
>>> and
>>> >>>>>>>>>> then
>>> >>>>>>>>>> cherry-picking it onto a new branch that has the
>>> hyracks+asterix
>>> >>>>>>>>>> master
>>> >>>>>>>>>> as
>>> >>>>>>>>>> the head.
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Wed, Mar 30, 2016 at 5:42 PM, Till Westmann <
>>> tillw@apache.org>
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> To get existing patches in, could we just create a textual
>>> patch
>>> >>>>>>>>>> (e.g.
>>> >>>>>>>>>>
>>> >>>>>>>>>> from gerrit), apply that with the necessary -p option to a n=
ew
>>> >>>>>>>>>>> local
>>> >>>>>>>>>>> checkout of the merged repositories and submit a new review
>>> to
>>> >>>>>>>>>>> gerrit?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Thanks,
>>> >>>>>>>>>>> Till
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On 30 Mar 2016, at 12:36, Ian Maxon wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Hi all,
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> I went ahead preliminarily merged the Hyracks and AsterixDB
>>> >>>>>>>>>>>> repositories
>>> >>>>>>>>>>>> into one. Unfortunately this can't be reviewed in Gerrit s=
o
>>> you
>>> >>>>>>>>>>>> all can
>>> >>>>>>>>>>>> check it out here:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> https://github.com/parshimers/incubator-asterixdb/tree/imaxon/merge-hyr=
acks
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> You will likely have to do some ugly rebasing for whatever
>>> >>>>>>>>>>>> changes
>>> >>>>>>>>>>>> you
>>> >>>>>>>>>>>> might have open once this gets done, since it moves
>>> asterixdb
>>> >>>>>>>>>>>> down
>>> >>>>>>>>>>>> one
>>> >>>>>>>>>>>> folder and swaps out pom.xml in the repository root.
>>> Hyracks is
>>> >>>>>>>>>>>> in
>>> >>>>>>>>>>>> a
>>> >>>>>>>>>>>> similar situation, though you would want to reapply your
>>> change
>>> >>>>>>>>>>>> to
>>> >>>>>>>>>>>> the
>>> >>>>>>>>>>>> AsterixDB repo from Hyracks (which is a bit odd). If you
>>> would
>>> >>>>>>>>>>>> like to
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> see
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>> how this affects your branch please do try fetching the
>>> branch I
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> linked
>>> >>>>>>>>>>>> above and testing it out on a copy of your topic branch.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> I'm still making sure all of the tests pass but nothing's
>>> >>>>>>>>>>>> failed so
>>> >>>>>>>>>>>> far.
>>> >>>>>>>>>>>> Unless anyone has objections I think we should push this
>>> change
>>> >>>>>>>>>>>> either
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> this
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>> week or early next week.
>>> >>>>>>>>>>>
>>> >>>>>>>>>>>> Let me know what you all think.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Thanks,
>>> >>>>>>>>>>>> - Ian
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>
>>> >
>>>
>>
>>
>

--001a113d2d662a7cc7052fe75a67--