asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Migration of git repository
Date Mon, 08 Jun 2015 15:00:09 GMT
These are great considerations and discussion.  The super key question for
me is the audience.



On Mon, Jun 8, 2015 at 11:08 AM, Chris Hillery <chillery@hillery.land>
wrote:

> If not, it may be worth taking a step back and asking what exactly the
> problem is. I understand the general rule that "we don't want Asterix to be
> broken", but what precisely does that mean? Is it acceptable that the tip
> of the Asterix source branch is only guaranteed to build against the tip of
> the Hyracks branch, for example? If not, why not? What audience are we
> required to keep things working for at the source level, and what
> expectations do they have?
>
> Ceej
> aka Chris Hillery
>
> On Mon, Jun 8, 2015 at 2:06 AM, Chris Hillery <chillery@hillery.land>
> wrote:
>
> > So, if we pushed these not-releases to the Nexus repo running at UCI, and
> > devs pulled from there in preference to "official" repos, that would
> solve
> > the problem?
> >
> > Ceej
> > aka Chris Hillery
> >
> > On Sun, Jun 7, 2015 at 7:29 PM, Ted Dunning <ted.dunning@gmail.com>
> wrote:
> >
> >>
> >> If it is pushed to any wider audience than roughly the dev@ list, it is
> >> a release. That definitely includes maven central.  Artifacts in maven
> are
> >> convenience binaries and this not a release but they should be
> traceable to
> >> an exact source release.
> >>
> >> Sent from my iPhone
> >>
> >> > On Jun 7, 2015, at 19:10, Till Westmann <tillw@apache.org> wrote:
> >> >
> >> > Hmm, good point. It doesn’t have to. One question might be if we can
> >> push it to some maven repository, if it’s not an official release.
> >> > But I think that should also be fine as long as we don’t push it to a
> >> repository that claims to contain official releases.
> >> >
> >> > Some mentor input might be helpful on this as well :)
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> >> On Jun 7, 2015, at 6:53 PM, Ildar Absalyamov <
> >> ildar.absalyamov@gmail.com> wrote:
> >> >>
> >> >> Does version bump always mean full-fledged Apache release? We need
> the
> >> former just to resolve compile time dependencies.
> >> >>
> >> >>> On Jun 7, 2015, at 18:49, Till Westmann <tillw@apache.org>
wrote:
> >> >>>
> >> >>> In principle I agree with this, but creating a new release will
be a
> >> little more involved that just running maven, when we do this at the
> ASF.
> >> >>> To publish a new release we will have to vet and vote on the
> release.
> >> This takes at least 72 hours  in the best case if we’re a TLP, the first
> >> release candidate is great, and have enough people to vote. While we’re
> >> still in the incubator, releasing will take a little longer as we also
> have
> >> to get enough votes for the release in the incubator.
> >> >>> As I proposed earlier, it would be really good to go through the
> full
> >> release process once, before we decide how to structure our processes
> and
> >> infrastructure.
> >> >>>
> >> >>> Cheers,
> >> >>> Till
> >> >>>
> >> >>>> On Jun 4, 2015, at 6:37 PM, Ildar Absalyamov <
> >> ildar.absalyamov@gmail.com> wrote:
> >> >>>>
> >> >>>> I am with Chris on repository separation and I think that the
> >> solution to the issue of Hyracks commits breaking Asterix build is using
> >> release Hyracks versions instead of snapshot ones. Yes, that will
> create a
> >> frequent Hyracks releases (we will have to release it each time there
> is a
> >> change which spans both Hyracks & Asterix) and we have abandoned this
> >> practice a while ago, but it seems that’s the only way to separate
> projects
> >> logically.
> >> >>>>
> >> >>>> Here are few examples to clear the picture. In all examples
Hyracks
> >> version is 4.5.6-Snapshot, Asterix version is 1.2.3-Snapshot (but it
> >> depends on previous release version Hyracks 4.5.5):
> >> >>>> 1) The changes span both Asterix & Hyracks.
> >> >>>> First make sure that Asterix could depend on Hyracks 4.5.6-Snapshot
> >> without API conflicts & switch Asterix dependency to 4.5.6-Snapshot.
> >> >>>> Submit Gerrit review, once it is done as a part of git-asf
script
> >> commit changes, bump Hyracks version to 4.5.6, make Asterix depend on
> 4.5.6
> >> and bump Hyracks to 4.5.7-Snapshot right after.
> >> >>>> 2) The changes are located only in Hyracks. Regular review
and
> >> commit (with snapshot version) without any version bump.
> >> >>>> 3) The changes are located only in Asterix. Regular review
and
> >> commit (with snapshot version) without any version bump.
> >> >>>>
> >> >>>> In this scenario Hyracks commit can never make Asterix build
fail
> >> (since it depends on a stable release) and it’s the responsibility of
> the
> >> first person, whose commits spans both repos to make sure that the
> changes
> >> in snapshot Hyracks version are properly merged.
> >> >>>>
> >> >>>> Regarding the Yingyi’s issue with Gerrit topics: could we
modify
> >> git-gerrit script so it would submit both Asterix & Hyracks reviews
> >> (granted that the latter is needed), and link them together, setting the
> >> proper topic? Gerrit seems to have API for changing that, right?
> >> >>>>
> >> >>>>> On Jun 4, 2015, at 15:45, Mike Carey <dtabass@gmail.com>
wrote:
> >> >>>>>
> >> >>>>> Just a quick high-level note from our nearest equivalent
of the
> >> pointy-haired Dilbert guy (aka me):  What would be nice is to have
> Hyracks
> >> changes kick off tests of all "supported client projects" - AsterixDB,
> >> VXQuery, maybe also Pregelix, IMRU, and possibly others in the future.
> I
> >> don't think we'll ever prevent such downstream things from being broken
> >> unless we run their tests - so I would suggest that we need a mechanism
> to
> >> keep Hyracks changes from being permitted to happen without verifying
> the
> >> ongoing integrity of all "blessed" (priority 1) affected projects....
> We
> >> could have an agreed upon list of such projects and tests for each....
> It
> >> would be nice to have a "quick check" (hello world still works, basics
> are
> >> working) that was synchronously blocking of such changes, and at least a
> >> daily verification that all's totally well (AFAWK) for them all.
> >> >>>>>
> >> >>>>> Not sure how this affects the still two-sided discussion...
 :-)
> >> >>>>>
> >> >>>>> Cheers,
> >> >>>>> Mike
> >> >>>>>
> >> >>>>>
> >> >>>>>> On 6/2/15 10:00 AM, Chris Hillery wrote:
> >> >>>>>>> On Mon, Jun 1, 2015 at 9:46 PM, Yingyi Bu <buyingyi@gmail.com>
> >> wrote:
> >> >>>>>>>
> >> >>>>>>> In my opinion,  merging the repository doesn't
break the
> >> separation of
> >> >>>>>>> hyracks and asterixdb, because the dependencies
are controlled
> by
> >> mvn pom
> >> >>>>>>> files.
> >> >>>>>>>
> >> >>>>>> That wasn't the separation I was talking about. I meant
API
> >> separation. As
> >> >>>>>> it is now, when we make a change to both Asterix and
Hyracks, we
> >> are forced
> >> >>>>>> to consider the API implications, or at least they
are put out
> >> there in a
> >> >>>>>> very clear way that we need to look at. If we merge
them, people
> >> will
> >> >>>>>> (rightly) treat the whole thing as one product, and
there will be
> >> no brakes
> >> >>>>>> on making wide-ranging API changes.
> >> >>>>>>
> >> >>>>>> (As an aside: I don't trust Maven's pom files to do
a good job of
> >> keeping
> >> >>>>>> the dependency management clean. In fact I trust it
to do
> >> precisely the
> >> >>>>>> opposite, by making it both easier to screw up the
dependencies
> >> and harder
> >> >>>>>> to update them in future.)
> >> >>>>>>
> >> >>>>>> Again, my point is this: If we truly believe that Hyracks
is a
> >> re-usable
> >> >>>>>> component, it should be treated as such from source
to build to
> >> delivery.
> >> >>>>>> By merging in Asterix, we are saying that Asterix is
"more equal"
> >> than
> >> >>>>>> others Hyracks clients, to the point that we're tacitly
willing
> to
> >> break
> >> >>>>>> those other clients in favor of simplifying Asterix
development.
> >> If that is
> >> >>>>>> a fair and true statement, well, then, sure, let's
merge them.
> >> >>>>>>
> >> >>>>>> 1) It forces those hyracks-only changes to pass asterixdb
> >> regression
> >> >>>>>>> tests.  Currently hyracks-only change are not verified
by
> >> asterixdb tests.
> >> >>>>>>>
> >> >>>>>> This is a good point, I will admit. However, I think
this same
> >> goal can be
> >> >>>>>> met in other ways. My strong preference would be to
create a set
> >> of true
> >> >>>>>> API tests inside of Hyracks, which both document and
test the
> >> external
> >> >>>>>> Hyracks API. That will make API-breaking changes in
future much
> >> easier to
> >> >>>>>> spot, and also make it clear when Asterix is using
internal APIs
> >> that it
> >> >>>>>> should not.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>> 2) On my local machine,  I don't need to always
install hyracks
> >> and then
> >> >>>>>>> verify asterixdb from time to time.  Especially,
switching
> >> branches seems
> >> >>>>>>> painful because the installed hyracks snapshot
is overwritten
> >> from time to
> >> >>>>>>> time.
> >> >>>>>>>
> >> >>>>>> I haven't tried working on multiple Hyracks branches
at the same
> >> time, so I
> >> >>>>>> haven't experienced this. This seems like a working
method error,
> >> though.
> >> >>>>>> If you're working with two things that are "the same
version"
> >> (even if
> >> >>>>>> that's a snapshot version), you'll need to use separate
Maven
> >> repositories
> >> >>>>>> to install them. In fact, merging the two git repositories
would
> >> do nothing
> >> >>>>>> to fix this problem, will it? If the proposal is to
put the two
> >> source
> >> >>>>>> repositories in the same git repo but otherwise leave
them
> >> untouched, then
> >> >>>>>> nothing would change in the build process. It's possible
I'm
> >> missing
> >> >>>>>> something there, though.
> >> >>>>>>
> >> >>>>>>
> >> >>>>>>> 3) I only need to make one code review request
and one jenkins
> >> job.
> >> >>>>>>> Currently I need to manually change the topic of
my asterixdb
> >> gerrit CL
> >> >>>>>>> every time before I update my hyracks CL, and then
manually
> >> schedule
> >> >>>>>>> jenkins to run a new asterixdb job.  If I forget
to schedule the
> >> jenkins
> >> >>>>>>> job, the asterixdb CL is still shown to be "verified
by
> jenkins".
> >> >>>>>>>
> >> >>>>>> This is a problem, but it's a problem in commit validation,
not
> in
> >> the
> >> >>>>>> source. Modifying the source to work around these issues
is still
> >> a bad
> >> >>>>>> idea IMHO.
> >> >>>>>>
> >> >>>>>> The "change-topic" issue could be fixed with a bit
of development
> >> work
> >> >>>>>> (have the topic point to a change, rather than a specific
> patchset
> >> on the
> >> >>>>>> change, so you only need to set it once, for instance).
> >> >>>>>>
> >> >>>>>> As for manually scheduling Asterix Jenkins jobs, that
sounds like
> >> it's only
> >> >>>>>> a problem where your Hyracks change breaks an existing
public
> API.
> >> That
> >> >>>>>> would be obviated by having true API testing inside
of Hyracks,
> >> which is
> >> >>>>>> something that we should have regardless of any decisions
about
> >> source
> >> >>>>>> locations.
> >> >>>>>>
> >> >>>>>> In summary / repeating myself again: yes, we have some
problems
> >> because
> >> >>>>>> Hyracks and Asterix are in seperate repositories. But
those
> >> problems are
> >> >>>>>> pointing out true issues with our development and processes.
> >> Merging the
> >> >>>>>> repositories isn't fixing those problems, it's sweeping
them
> under
> >> the rug.
> >> >>>>>> Long term we would be much better off to identify,
isolate, and
> >> fix the
> >> >>>>>> problems themselves.
> >> >>>>>>
> >> >>>>>> Ceej
> >> >>>>>> aka Chris Hillery
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>> Best regards,
> >> >>>> Ildar
> >> >>>>
> >> >>>
> >> >>
> >> >> Best regards,
> >> >> Ildar
> >> >>
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message