Mailing-List: contact dev-help@hawq.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hawq.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <CABQrizfH_oQSNHdBPKODWLSf+j6J50UzuNmhOap8cb07zGN5fg@mail.gmail.com>
References: <CABQrizdJJMQY6vFLJ0_qcRHw-BH3QcQmTOSv0KoC-9Y8PfHiOA@mail.gmail.com>
 <CACkst5C5xQL1+2tKS0PmoiVHz7P05mk0-gOQhazT=q8zAEk-gQ@mail.gmail.com>
 <CAFht1h+kmf0E1-xkRbArBwpsmif7-a8LRbc-9zLy=OtTvXWoVQ@mail.gmail.com>
 <CAH=zMbpbOoxbfX2NQ4n12m0xJoeTRrtiKCq3LtLq_-6igNB6AA@mail.gmail.com>
 <CAA=Rw+HiDx8djJpndpMK+F9TswGhyNZ-audmJewc=zV2tXOQww@mail.gmail.com>
 <CABQrizesjCOGEK2QqyP_XmgJ5kqCEnbYZz6u3Y4dQTzj7NBa4g@mail.gmail.com>
 <CABQrizfn1ZWx-Np-U0EO_TeOuj+dnNTWv6Stu2LC4o0sZe8fZw@mail.gmail.com>
 <CA+ULb+to+QC_Eo0Q++_QtLLQGSb0hSd2ZikEXDHfL0hzuBcLFA@mail.gmail.com>
 <CAH=zMbozdiUqE30wFM6akHa1eb2_AejfQPq_jzjLhSV6f=Yb9w@mail.gmail.com> <CABQrizfH_oQSNHdBPKODWLSf+j6J50UzuNmhOap8cb07zGN5fg@mail.gmail.com>
From: Guo Gang <paulguo@gmail.com>
Date: Fri, 1 Jul 2016 15:47:28 +0800
Message-ID: <CABQrizfOL6yY6H1osdcO9W_Q6oANZ2tihRd9H3Xk6gtJ7P3d3A@mail.gmail.com>
Subject: Re: Replace git submodule with git clone + file with commit number?
To: dev@hawq.incubator.apache.org
Content-Type: multipart/alternative; boundary=001a11343536d177d005368e3253
archived-at: Fri, 01 Jul 2016 07:47:37 -0000

--001a11343536d177d005368e3253
Content-Type: text/plain; charset=UTF-8

Radar, it seems that the maven issue could be resolved similarly?

2016-07-01 15:31 GMT+08:00 Guo Gang <paulguo@gmail.com>:

> Those submodules will be git-cloned and then compiled only with specific
> options
> are enabled in configure command. If users want to build them without
> Internet
> connection they will have to download related git repo with specific
> commit number in advance and copy them to hawq repo. Or they can set up
> internal git repos and modify the git addresses in hawq source.
>
>
>
> 2016-07-01 14:07 GMT+08:00 Radar Da lei <rlei@pivotal.io>:
>
>> One thought is if they all have released source tarball, we can let user
>> download these tarballs to a specified directory.
>>
>> Then when compiling, we do not do 'git clone' but extract these tarballs
>> if
>> the specified version tarballs are detected.
>>
>> But for user who don't have internet access, when compiling, maven
>> download
>> would be a problem. See:
>>
>> Downloading:
>>
>> https://repo.maven.apache.org/maven2/com/pivotal/hawq/hawq-mapreduce-ao/1.1.0/hawq-mapreduce-ao-1.1.0.jar
>> Downloading:
>>
>> https://repo.maven.apache.org/maven2/com/pivotal/hawq/hawq-mapreduce-parquet/1.1.0/hawq-mapreduce-parquet-1.1.0.jar
>> [INFO] hawq-hadoop ........................................ FAILURE [
>>  1.757 s]
>>
>> Thanks.
>>
>> Regards,
>> Radar
>>
>> On Fri, Jul 1, 2016 at 12:30 AM, Roman Shaposhnik <roman@shaposhnik.org>
>> wrote:
>>
>> > Good discussion so far, but I wanted to make sure that our source
>> > tarball has an option of being built in an environment completely
>> > disconnected from the internet (you can mandate certain shared
>> > librarires to be available in the environment, of course). Will this
>> > be possible with the current proposal?
>> >
>> > Thanks,
>> > Roman.
>> >
>> > On Thu, Jun 30, 2016 at 3:51 AM, Guo Gang <paulguo@gmail.com> wrote:
>> > > Discussed more with Jiali, I realised that to make hawk repo be more
>> > > friendly to gporca developers, I could store git address, branch and a
>> > > commit number in one file so that gporca developers could easier
>> switch
>> > > gporca code.
>> > >
>> > > 2016-06-30 15:35 GMT+08:00 Guo Gang <paulguo@gmail.com>:
>> > >
>> > >> 1. HAWQ repo is not a good place for gp-xerces development. My change
>> > just
>> > >>    use a stable and working version of gp-xerces. If gp-xerces
>> > developers
>> > >> want to
>> > >>    test it with hawq, they can easily modify code under hawk:
>> gp-xerces
>> > >> (e.g. pull from
>> > >>     a branch, etc.), then remove the timestamp file
>> > >> depends/thirdpaty/gp-xerces_build_timestamp
>> > >>    to trigger a rebuild.
>> > >>
>> > >> 2. Correct.
>> > >>
>> > >> 2016-06-30 11:31 GMT+08:00 Jiali Yao <jyao@pivotal.io>:
>> > >>
>> > >>> Hi Paul
>> > >>>
>> > >>> Generally agree on the approach while I have some questions:
>> > >>>
>> > >>> 1. Do we need to add branch handling?  For example, I am a developer
>> > >>> of gp-xerces,
>> > >>>  I need to test my branch code.
>> > >>>
>> > >>> 2. Let me double confirm about the build process.  Below is
>> gp-xerces
>> > repo
>> > >>> and now HAWQ will be commit a. So I will set commit file
>> > gp-xerces.commit
>> > >>> in HAWQ repo which indicate we will use commit a. And then sometime
>> > later,
>> > >>> we want to use b, then we commit   gp-xerces.commit to b right?
>> And we
>> > >>> will have multiple .commit file for each submodule now. right?
>> > >>> gp-xerces:
>> > >>>      commit c
>> > >>>      ....
>> > >>>      commit b
>> > >>>      ....
>> > >>>      commit a
>> > >>>
>> > >>> Thanks
>> > >>> Jiali
>> > >>>
>> > >>> On Thu, Jun 30, 2016 at 11:03 AM, Radar Da lei <rlei@pivotal.io>
>> > wrote:
>> > >>>
>> > >>> > +1
>> > >>> > Since 'git clone' is called automatically in Makefile, I feel
>> this is
>> > >>> much
>> > >>> > better than use submodule.
>> > >>> >
>> > >>> > Regards,
>> > >>> > Radar
>> > >>> >
>> > >>> > On Thu, Jun 30, 2016 at 10:07 AM, Lili Ma <lma@pivotal.io> wrote:
>> > >>> >
>> > >>> > > Hi Paul,
>> > >>> > >
>> > >>> > > I have one question, you mentioned "if I update gp-xerces commit
>> > >>> number
>> > >>> > in
>> > >>> > > the commit file gp-xerces.commit make will trigger an
>> auto-build".
>> > >>> Could
>> > >>> > > we set the $(gp_xerces_commit) to any commit number? What I mean
>> > is,
>> > >>> > could
>> > >>> > > we set it to a commit number already existing in my local code,
>> or
>> > >>> from
>> > >>> > > remote code which I have not fetched?
>> > >>> > >
>> > >>> > > If the answer is yes, I totally agree this suggestion.
>> > >>> > >
>> > >>> > > Thanks
>> > >>> > > Lili
>> > >>> > >
>> > >>> > > On Wed, Jun 29, 2016 at 11:52 PM, Kavinder Dhaliwal <
>> > >>> > kdhaliwal@pivotal.io>
>> > >>> > > wrote:
>> > >>> > >
>> > >>> > > > +1. I am in favor of this approach, especially if the
>> submodules
>> > >>> make
>> > >>> > the
>> > >>> > > > source tarball difficult to build. I also agree it will make a
>> > >>> > developers
>> > >>> > > > life much easier.
>> > >>> > > >
>> > >>> > > > On Wed, Jun 29, 2016 at 3:18 AM, Guo Gang <paulguo@gmail.com>
>> > >>> wrote:
>> > >>> > > >
>> > >>> > > > > I'm proposing this change because:
>> > >>> > > > >
>> > >>> > > > > 1) We are ready to release the first apache release with
>> > "source
>> > >>> > > > tarball",
>> > >>> > > > > but
>> > >>> > > > >     submodule is not friendly to source tarball since git
>> > >>> submodule
>> > >>> > > > >    require a git parent.
>> > >>> > > > >
>> > >>> > > > > 2) With more and more development, I found that submodule
>> > >>> mechanism
>> > >>> > > > >     is not that friendly for development, e.g.
>> > >>> > > > >
>> > >>> > > > >     If the commit number of one submodule is modified, it is
>> > hard
>> > >>> to
>> > >>> > > > detect
>> > >>> > > > >     this for Makefile. We need to manually update it in an
>> old
>> > >>> repo.
>> > >>> > If
>> > >>> > > > >     using "git clone", we could easily detect update by set
>> a
>> > >>> commit
>> > >>> > > > number
>> > >>> > > > > file
>> > >>> > > > >     as dependency and thus easily update the code.
>> > >>> > > > >
>> > >>> > > > >    Some developers have complained the annoying output in
>> "git
>> > >>> > status"
>> > >>> > > > >    after building submodules. With "git clone" we can easily
>> > mask
>> > >>> > those
>> > >>> > > > >    directories via .gitignore.
>> > >>> > > > >
>> > >>> > > > >    It is hard for developers who is not familiar with
>> submodule
>> > >>> > > (Frankly
>> > >>> > > > > speaking
>> > >>> > > > >    I really do not think submodule is friendly) to
>> manipulate
>> > >>> related
>> > >>> > > > > directories
>> > >>> > > > >    when necessary.
>> > >>> > > > >
>> > >>> > > > > While with the "git clone" solution, we save the commit of
>> > >>> previous
>> > >>> > > > > submodule in a file,
>> > >>> > > > > and the related Makefile code change is rather simple, e.g.
>> for
>> > >>> > > > gp-xerces.
>> > >>> > > > >
>> > >>> > > > > $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp:
>> > >>> > > > > $(ORCA_SRC_PATH)/gp-xerces.commit
>> > >>> > > > >     rm -f $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
>> > >>> > > > >
>> > >>> > > > > gp-xerces_prepare:
>> $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
>> > >>> > > > >     if [ ! -f $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
>> ];
>> > >>> then \
>> > >>> > > > >         [ "x$(gp_xerces_commit)" != "x" ] || exit 1; \
>> > >>> > > > >         cd $(abs_top_srcdir)/$(subdir); mkdir -p gp-xerces;
>> cd
>> > >>> > > > gp-xerces; \
>> > >>> > > > >         [ ! -d .git ] && git clone
>> > >>> > > > > https://github.com/greenplum-db/gp-xerces.git .; \
>> > >>> > > > >         git reset --hard $(gp_xerces_commit) || exit 2; \
>> > >>> > > > >         touch $(ORCA_BLD_PATH)/gp_xerces_prepare_timestamp;
>> \
>> > >>> > > > >     fi
>> > >>> > > > >
>> > >>> > > > > With above code change, if I update gp-xerces commit number
>> in
>> > the
>> > >>> > > commit
>> > >>> > > > > file
>> > >>> > > > > gp-xerces.commit make will trigger an auto-build, If I
>> messed
>> > up
>> > >>> the
>> > >>> > > > > gp-xerces directory
>> > >>> > > > > I can easily remove the whole gp-xerces, or just remove the
>> > >>> timestamp
>> > >>> > > > file
>> > >>> > > > > gp-xerces_prepare_timestamp to trigger an auto build.
>> > >>> > > > >
>> > >>> > > > > Any suggestion? Thanks.
>> > >>> > > > >
>> > >>> > > >
>> > >>> > >
>> > >>> >
>> > >>>
>> > >>
>> > >>
>> >
>>
>
>

--001a11343536d177d005368e3253--