hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radar Da lei <r...@pivotal.io>
Subject Re: Replace git submodule with git clone + file with commit number?
Date Fri, 01 Jul 2016 06:07:44 GMT
One thought is if they all have released source tarball, we can let user
download these tarballs to a specified directory.

Then when compiling, we do not do 'git clone' but extract these tarballs if
the specified version tarballs are detected.

But for user who don't have internet access, when compiling, maven download
would be a problem. See:

Downloading:
https://repo.maven.apache.org/maven2/com/pivotal/hawq/hawq-mapreduce-ao/1.1.0/hawq-mapreduce-ao-1.1.0.jar
Downloading:
https://repo.maven.apache.org/maven2/com/pivotal/hawq/hawq-mapreduce-parquet/1.1.0/hawq-mapreduce-parquet-1.1.0.jar
[INFO] hawq-hadoop ........................................ FAILURE [
 1.757 s]

Thanks.

Regards,
Radar

On Fri, Jul 1, 2016 at 12:30 AM, Roman Shaposhnik <roman@shaposhnik.org>
wrote:

> Good discussion so far, but I wanted to make sure that our source
> tarball has an option of being built in an environment completely
> disconnected from the internet (you can mandate certain shared
> librarires to be available in the environment, of course). Will this
> be possible with the current proposal?
>
> Thanks,
> Roman.
>
> On Thu, Jun 30, 2016 at 3:51 AM, Guo Gang <paulguo@gmail.com> wrote:
> > Discussed more with Jiali, I realised that to make hawk repo be more
> > friendly to gporca developers, I could store git address, branch and a
> > commit number in one file so that gporca developers could easier switch
> > gporca code.
> >
> > 2016-06-30 15:35 GMT+08:00 Guo Gang <paulguo@gmail.com>:
> >
> >> 1. HAWQ repo is not a good place for gp-xerces development. My change
> just
> >>    use a stable and working version of gp-xerces. If gp-xerces
> developers
> >> want to
> >>    test it with hawq, they can easily modify code under hawk: gp-xerces
> >> (e.g. pull from
> >>     a branch, etc.), then remove the timestamp file
> >> depends/thirdpaty/gp-xerces_build_timestamp
> >>    to trigger a rebuild.
> >>
> >> 2. Correct.
> >>
> >> 2016-06-30 11:31 GMT+08:00 Jiali Yao <jyao@pivotal.io>:
> >>
> >>> Hi Paul
> >>>
> >>> Generally agree on the approach while I have some questions:
> >>>
> >>> 1. Do we need to add branch handling?  For example, I am a developer
> >>> of gp-xerces,
> >>>  I need to test my branch code.
> >>>
> >>> 2. Let me double confirm about the build process.  Below is gp-xerces
> repo
> >>> and now HAWQ will be commit a. So I will set commit file
> gp-xerces.commit
> >>> in HAWQ repo which indicate we will use commit a. And then sometime
> later,
> >>> we want to use b, then we commit   gp-xerces.commit to b right?  And we
> >>> will have multiple .commit file for each submodule now. right?
> >>> gp-xerces:
> >>>      commit c
> >>>      ....
> >>>      commit b
> >>>      ....
> >>>      commit a
> >>>
> >>> Thanks
> >>> Jiali
> >>>
> >>> On Thu, Jun 30, 2016 at 11:03 AM, Radar Da lei <rlei@pivotal.io>
> wrote:
> >>>
> >>> > +1
> >>> > Since 'git clone' is called automatically in Makefile, I feel this
is
> >>> much
> >>> > better than use submodule.
> >>> >
> >>> > Regards,
> >>> > Radar
> >>> >
> >>> > On Thu, Jun 30, 2016 at 10:07 AM, Lili Ma <lma@pivotal.io> wrote:
> >>> >
> >>> > > Hi Paul,
> >>> > >
> >>> > > I have one question, you mentioned "if I update gp-xerces commit
> >>> number
> >>> > in
> >>> > > the commit file gp-xerces.commit make will trigger an auto-build".
> >>> Could
> >>> > > we set the $(gp_xerces_commit) to any commit number? What I mean
> is,
> >>> > could
> >>> > > we set it to a commit number already existing in my local code,
or
> >>> from
> >>> > > remote code which I have not fetched?
> >>> > >
> >>> > > If the answer is yes, I totally agree this suggestion.
> >>> > >
> >>> > > Thanks
> >>> > > Lili
> >>> > >
> >>> > > On Wed, Jun 29, 2016 at 11:52 PM, Kavinder Dhaliwal <
> >>> > kdhaliwal@pivotal.io>
> >>> > > wrote:
> >>> > >
> >>> > > > +1. I am in favor of this approach, especially if the submodules
> >>> make
> >>> > the
> >>> > > > source tarball difficult to build. I also agree it will make
a
> >>> > developers
> >>> > > > life much easier.
> >>> > > >
> >>> > > > On Wed, Jun 29, 2016 at 3:18 AM, Guo Gang <paulguo@gmail.com>
> >>> wrote:
> >>> > > >
> >>> > > > > I'm proposing this change because:
> >>> > > > >
> >>> > > > > 1) We are ready to release the first apache release
with
> "source
> >>> > > > tarball",
> >>> > > > > but
> >>> > > > >     submodule is not friendly to source tarball since
git
> >>> submodule
> >>> > > > >    require a git parent.
> >>> > > > >
> >>> > > > > 2) With more and more development, I found that submodule
> >>> mechanism
> >>> > > > >     is not that friendly for development, e.g.
> >>> > > > >
> >>> > > > >     If the commit number of one submodule is modified,
it is
> hard
> >>> to
> >>> > > > detect
> >>> > > > >     this for Makefile. We need to manually update it
in an old
> >>> repo.
> >>> > If
> >>> > > > >     using "git clone", we could easily detect update
by set a
> >>> commit
> >>> > > > number
> >>> > > > > file
> >>> > > > >     as dependency and thus easily update the code.
> >>> > > > >
> >>> > > > >    Some developers have complained the annoying output
in "git
> >>> > status"
> >>> > > > >    after building submodules. With "git clone" we can
easily
> mask
> >>> > those
> >>> > > > >    directories via .gitignore.
> >>> > > > >
> >>> > > > >    It is hard for developers who is not familiar with
submodule
> >>> > > (Frankly
> >>> > > > > speaking
> >>> > > > >    I really do not think submodule is friendly) to manipulate
> >>> related
> >>> > > > > directories
> >>> > > > >    when necessary.
> >>> > > > >
> >>> > > > > While with the "git clone" solution, we save the commit
of
> >>> previous
> >>> > > > > submodule in a file,
> >>> > > > > and the related Makefile code change is rather simple,
e.g. for
> >>> > > > gp-xerces.
> >>> > > > >
> >>> > > > > $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp:
> >>> > > > > $(ORCA_SRC_PATH)/gp-xerces.commit
> >>> > > > >     rm -f $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
> >>> > > > >
> >>> > > > > gp-xerces_prepare: $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
> >>> > > > >     if [ ! -f $(ORCA_BLD_PATH)/gp-xerces_prepare_timestamp
];
> >>> then \
> >>> > > > >         [ "x$(gp_xerces_commit)" != "x" ] || exit 1;
\
> >>> > > > >         cd $(abs_top_srcdir)/$(subdir); mkdir -p gp-xerces;
cd
> >>> > > > gp-xerces; \
> >>> > > > >         [ ! -d .git ] && git clone
> >>> > > > > https://github.com/greenplum-db/gp-xerces.git .; \
> >>> > > > >         git reset --hard $(gp_xerces_commit) || exit
2; \
> >>> > > > >         touch $(ORCA_BLD_PATH)/gp_xerces_prepare_timestamp;
\
> >>> > > > >     fi
> >>> > > > >
> >>> > > > > With above code change, if I update gp-xerces commit
number in
> the
> >>> > > commit
> >>> > > > > file
> >>> > > > > gp-xerces.commit make will trigger an auto-build, If
I messed
> up
> >>> the
> >>> > > > > gp-xerces directory
> >>> > > > > I can easily remove the whole gp-xerces, or just remove
the
> >>> timestamp
> >>> > > > file
> >>> > > > > gp-xerces_prepare_timestamp to trigger an auto build.
> >>> > > > >
> >>> > > > > Any suggestion? Thanks.
> >>> > > > >
> >>> > > >
> >>> > >
> >>> >
> >>>
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message