asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Maxon <ima...@uci.edu>
Subject Re: Migration of git repository
Date Mon, 08 Jun 2015 16:39:44 GMT
This could be workable, but we can't use SNAPSHOT suffixed versions in the
typical way. There needs to be a version for every git commit, basically.
That is honestly the entire reason this is an issue. Maven has one idea of
versions, while we kind of treat git's revisions as versions of their own.

-Ian

On Mon, Jun 8, 2015 at 7:58 AM, Ted Dunning <ted.dunning@gmail.com> wrote:

> An internal development hot-patch (especially if marked SNAPSHOT) is fine.
> A publicly available version, especially if the info is disseminated, is
> not.
>
>
>
> On Mon, Jun 8, 2015 at 11:06 AM, Chris Hillery <chillery@hillery.land>
> wrote:
>
> > So, if we pushed these not-releases to the Nexus repo running at UCI, and
> > devs pulled from there in preference to "official" repos, that would
> solve
> > the problem?
> >
> > Ceej
> > aka Chris Hillery
> >
> > On Sun, Jun 7, 2015 at 7:29 PM, Ted Dunning <ted.dunning@gmail.com>
> wrote:
> >
> > >
> > > If it is pushed to any wider audience than roughly the dev@ list, it
> is
> > a
> > > release. That definitely includes maven central.  Artifacts in maven
> are
> > > convenience binaries and this not a release but they should be
> traceable
> > to
> > > an exact source release.
> > >
> > > Sent from my iPhone
> > >
> > > > On Jun 7, 2015, at 19:10, Till Westmann <tillw@apache.org> wrote:
> > > >
> > > > Hmm, good point. It doesn’t have to. One question might be if we can
> > > push it to some maven repository, if it’s not an official release.
> > > > But I think that should also be fine as long as we don’t push it to
a
> > > repository that claims to contain official releases.
> > > >
> > > > Some mentor input might be helpful on this as well :)
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > >> On Jun 7, 2015, at 6:53 PM, Ildar Absalyamov <
> > > ildar.absalyamov@gmail.com> wrote:
> > > >>
> > > >> Does version bump always mean full-fledged Apache release? We need
> the
> > > former just to resolve compile time dependencies.
> > > >>
> > > >>> On Jun 7, 2015, at 18:49, Till Westmann <tillw@apache.org>
wrote:
> > > >>>
> > > >>> In principle I agree with this, but creating a new release will
be
> a
> > > little more involved that just running maven, when we do this at the
> ASF.
> > > >>> To publish a new release we will have to vet and vote on the
> release.
> > > This takes at least 72 hours  in the best case if we’re a TLP, the
> first
> > > release candidate is great, and have enough people to vote. While we’re
> > > still in the incubator, releasing will take a little longer as we also
> > have
> > > to get enough votes for the release in the incubator.
> > > >>> As I proposed earlier, it would be really good to go through the
> full
> > > release process once, before we decide how to structure our processes
> and
> > > infrastructure.
> > > >>>
> > > >>> Cheers,
> > > >>> Till
> > > >>>
> > > >>>> On Jun 4, 2015, at 6:37 PM, Ildar Absalyamov <
> > > ildar.absalyamov@gmail.com> wrote:
> > > >>>>
> > > >>>> I am with Chris on repository separation and I think that
the
> > > solution to the issue of Hyracks commits breaking Asterix build is
> using
> > > release Hyracks versions instead of snapshot ones. Yes, that will
> create
> > a
> > > frequent Hyracks releases (we will have to release it each time there
> is
> > a
> > > change which spans both Hyracks & Asterix) and we have abandoned this
> > > practice a while ago, but it seems that’s the only way to separate
> > projects
> > > logically.
> > > >>>>
> > > >>>> Here are few examples to clear the picture. In all examples
> Hyracks
> > > version is 4.5.6-Snapshot, Asterix version is 1.2.3-Snapshot (but it
> > > depends on previous release version Hyracks 4.5.5):
> > > >>>> 1) The changes span both Asterix & Hyracks.
> > > >>>> First make sure that Asterix could depend on Hyracks
> 4.5.6-Snapshot
> > > without API conflicts & switch Asterix dependency to 4.5.6-Snapshot.
> > > >>>> Submit Gerrit review, once it is done as a part of git-asf
script
> > > commit changes, bump Hyracks version to 4.5.6, make Asterix depend on
> > 4.5.6
> > > and bump Hyracks to 4.5.7-Snapshot right after.
> > > >>>> 2) The changes are located only in Hyracks. Regular review
and
> > commit
> > > (with snapshot version) without any version bump.
> > > >>>> 3) The changes are located only in Asterix. Regular review
and
> > commit
> > > (with snapshot version) without any version bump.
> > > >>>>
> > > >>>> In this scenario Hyracks commit can never make Asterix build
fail
> > > (since it depends on a stable release) and it’s the responsibility of
> the
> > > first person, whose commits spans both repos to make sure that the
> > changes
> > > in snapshot Hyracks version are properly merged.
> > > >>>>
> > > >>>> Regarding the Yingyi’s issue with Gerrit topics: could we
modify
> > > git-gerrit script so it would submit both Asterix & Hyracks reviews
> > > (granted that the latter is needed), and link them together, setting
> the
> > > proper topic? Gerrit seems to have API for changing that, right?
> > > >>>>
> > > >>>>> On Jun 4, 2015, at 15:45, Mike Carey <dtabass@gmail.com>
wrote:
> > > >>>>>
> > > >>>>> Just a quick high-level note from our nearest equivalent
of the
> > > pointy-haired Dilbert guy (aka me):  What would be nice is to have
> > Hyracks
> > > changes kick off tests of all "supported client projects" - AsterixDB,
> > > VXQuery, maybe also Pregelix, IMRU, and possibly others in the
> future.  I
> > > don't think we'll ever prevent such downstream things from being broken
> > > unless we run their tests - so I would suggest that we need a mechanism
> > to
> > > keep Hyracks changes from being permitted to happen without verifying
> the
> > > ongoing integrity of all "blessed" (priority 1) affected projects....
> We
> > > could have an agreed upon list of such projects and tests for each....
> > It
> > > would be nice to have a "quick check" (hello world still works, basics
> > are
> > > working) that was synchronously blocking of such changes, and at least
> a
> > > daily verification that all's totally well (AFAWK) for them all.
> > > >>>>>
> > > >>>>> Not sure how this affects the still two-sided discussion...
 :-)
> > > >>>>>
> > > >>>>> Cheers,
> > > >>>>> Mike
> > > >>>>>
> > > >>>>>
> > > >>>>>> On 6/2/15 10:00 AM, Chris Hillery wrote:
> > > >>>>>>> On Mon, Jun 1, 2015 at 9:46 PM, Yingyi Bu <buyingyi@gmail.com>
> > > wrote:
> > > >>>>>>>
> > > >>>>>>> In my opinion,  merging the repository doesn't
break the
> > > separation of
> > > >>>>>>> hyracks and asterixdb, because the dependencies
are controlled
> by
> > > mvn pom
> > > >>>>>>> files.
> > > >>>>>>>
> > > >>>>>> That wasn't the separation I was talking about. I
meant API
> > > separation. As
> > > >>>>>> it is now, when we make a change to both Asterix and
Hyracks, we
> > > are forced
> > > >>>>>> to consider the API implications, or at least they
are put out
> > > there in a
> > > >>>>>> very clear way that we need to look at. If we merge
them, people
> > > will
> > > >>>>>> (rightly) treat the whole thing as one product, and
there will
> be
> > > no brakes
> > > >>>>>> on making wide-ranging API changes.
> > > >>>>>>
> > > >>>>>> (As an aside: I don't trust Maven's pom files to do
a good job
> of
> > > keeping
> > > >>>>>> the dependency management clean. In fact I trust it
to do
> > precisely
> > > the
> > > >>>>>> opposite, by making it both easier to screw up the
dependencies
> > and
> > > harder
> > > >>>>>> to update them in future.)
> > > >>>>>>
> > > >>>>>> Again, my point is this: If we truly believe that
Hyracks is a
> > > re-usable
> > > >>>>>> component, it should be treated as such from source
to build to
> > > delivery.
> > > >>>>>> By merging in Asterix, we are saying that Asterix
is "more
> equal"
> > > than
> > > >>>>>> others Hyracks clients, to the point that we're tacitly
willing
> to
> > > break
> > > >>>>>> those other clients in favor of simplifying Asterix
development.
> > If
> > > that is
> > > >>>>>> a fair and true statement, well, then, sure, let's
merge them.
> > > >>>>>>
> > > >>>>>> 1) It forces those hyracks-only changes to pass asterixdb
> > regression
> > > >>>>>>> tests.  Currently hyracks-only change are not
verified by
> > > asterixdb tests.
> > > >>>>>>>
> > > >>>>>> This is a good point, I will admit. However, I think
this same
> > goal
> > > can be
> > > >>>>>> met in other ways. My strong preference would be to
create a set
> > of
> > > true
> > > >>>>>> API tests inside of Hyracks, which both document and
test the
> > > external
> > > >>>>>> Hyracks API. That will make API-breaking changes in
future much
> > > easier to
> > > >>>>>> spot, and also make it clear when Asterix is using
internal APIs
> > > that it
> > > >>>>>> should not.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> 2) On my local machine,  I don't need to always
install hyracks
> > > and then
> > > >>>>>>> verify asterixdb from time to time.  Especially,
switching
> > > branches seems
> > > >>>>>>> painful because the installed hyracks snapshot
is overwritten
> > from
> > > time to
> > > >>>>>>> time.
> > > >>>>>>>
> > > >>>>>> I haven't tried working on multiple Hyracks branches
at the same
> > > time, so I
> > > >>>>>> haven't experienced this. This seems like a working
method
> error,
> > > though.
> > > >>>>>> If you're working with two things that are "the same
version"
> > (even
> > > if
> > > >>>>>> that's a snapshot version), you'll need to use separate
Maven
> > > repositories
> > > >>>>>> to install them. In fact, merging the two git repositories
would
> > do
> > > nothing
> > > >>>>>> to fix this problem, will it? If the proposal is to
put the two
> > > source
> > > >>>>>> repositories in the same git repo but otherwise leave
them
> > > untouched, then
> > > >>>>>> nothing would change in the build process. It's possible
I'm
> > missing
> > > >>>>>> something there, though.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> 3) I only need to make one code review request
and one jenkins
> > job.
> > > >>>>>>> Currently I need to manually change the topic
of my asterixdb
> > > gerrit CL
> > > >>>>>>> every time before I update my hyracks CL, and
then manually
> > > schedule
> > > >>>>>>> jenkins to run a new asterixdb job.  If I forget
to schedule
> the
> > > jenkins
> > > >>>>>>> job, the asterixdb CL is still shown to be "verified
by
> jenkins".
> > > >>>>>>>
> > > >>>>>> This is a problem, but it's a problem in commit validation,
not
> in
> > > the
> > > >>>>>> source. Modifying the source to work around these
issues is
> still
> > a
> > > bad
> > > >>>>>> idea IMHO.
> > > >>>>>>
> > > >>>>>> The "change-topic" issue could be fixed with a bit
of
> development
> > > work
> > > >>>>>> (have the topic point to a change, rather than a specific
> patchset
> > > on the
> > > >>>>>> change, so you only need to set it once, for instance).
> > > >>>>>>
> > > >>>>>> As for manually scheduling Asterix Jenkins jobs, that
sounds
> like
> > > it's only
> > > >>>>>> a problem where your Hyracks change breaks an existing
public
> API.
> > > That
> > > >>>>>> would be obviated by having true API testing inside
of Hyracks,
> > > which is
> > > >>>>>> something that we should have regardless of any decisions
about
> > > source
> > > >>>>>> locations.
> > > >>>>>>
> > > >>>>>> In summary / repeating myself again: yes, we have
some problems
> > > because
> > > >>>>>> Hyracks and Asterix are in seperate repositories.
But those
> > > problems are
> > > >>>>>> pointing out true issues with our development and
processes.
> > > Merging the
> > > >>>>>> repositories isn't fixing those problems, it's sweeping
them
> under
> > > the rug.
> > > >>>>>> Long term we would be much better off to identify,
isolate, and
> > fix
> > > the
> > > >>>>>> problems themselves.
> > > >>>>>>
> > > >>>>>> Ceej
> > > >>>>>> aka Chris Hillery
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>> Best regards,
> > > >>>> Ildar
> > > >>>>
> > > >>>
> > > >>
> > > >> Best regards,
> > > >> Ildar
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message