Return-Path: X-Original-To: apmail-asterixdb-dev-archive@minotaur.apache.org Delivered-To: apmail-asterixdb-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 431DA1896D for ; Tue, 2 Jun 2015 04:34:45 +0000 (UTC) Received: (qmail 51109 invoked by uid 500); 2 Jun 2015 04:34:45 -0000 Delivered-To: apmail-asterixdb-dev-archive@asterixdb.apache.org Received: (qmail 51052 invoked by uid 500); 2 Jun 2015 04:34:45 -0000 Mailing-List: contact dev-help@asterixdb.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.incubator.apache.org Delivered-To: mailing list dev@asterixdb.incubator.apache.org Received: (qmail 51041 invoked by uid 99); 2 Jun 2015 04:34:45 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Jun 2015 04:34:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 976D4C0B58 for ; Tue, 2 Jun 2015 04:34:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.009 X-Spam-Level: X-Spam-Status: No, score=-0.009 tagged_above=-999 required=6.31 tests=[T_RP_MATCHES_RCVD=-0.01, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id zBD-rum8mJ6C for ; Tue, 2 Jun 2015 04:34:30 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with SMTP id B0D8623131 for ; Tue, 2 Jun 2015 04:34:28 +0000 (UTC) Received: (qmail 50725 invoked by uid 99); 2 Jun 2015 04:34:27 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Jun 2015 04:34:27 +0000 Received: from [192.168.1.81] (c-50-174-102-44.hsd1.ca.comcast.net [50.174.102.44]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id B38D61A010F; Tue, 2 Jun 2015 04:34:27 +0000 (UTC) From: "Till Westmann" To: "dev@asterixdb.incubator.apache.org" Subject: Re: Migration of git repository Date: Mon, 01 Jun 2015 21:34:16 -0700 Message-ID: <363B4918-BFB4-4CF9-BD4E-1D22205383C9@apache.org> In-Reply-To: References: <4D16706C-FB89-45F3-8909-D13A076696E5@westmann.org> <553671A7.7040801@ics.uci.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailer: MailMate (1.9.1r5084) I'm also not too fond about merging - especially with my VXQuery hat on. Another $.02, Till On 1 Jun 2015, at 17:15, Chris Hillery wrote: > My $.02 - no, we shouldn't. > > Two main reasons: > > 1. If we're serious about Hyracks being a re-usable component of other > products, it makes sense to dogfood that in Asterixdb. If there are > problems keeping Hyracks separate from Asterix or keeping Hyracks with > clean interfaces, this forces us to address them. > > 2. We only just recently took the initiative to take Pregelix and > Hiversterix *out* of the same repository, and that was because they > were > specifically causing us problems as components of the same build. > (There > were issues of competing dependency versions with Ian's YARN work, as > well > as several spurious pregelix test failures, as I recall.) At a bare > minimum, we cannot merge those projects back in without re-researching > and > addressing those problems. > > What benefits would we gain by merging them? I honestly don't agree > with > Yingyi's suggestion that it would make building, bug-fixing, and code > review much simpler. At best it would help a bit on those occasions > when a > change spans Hyracks and Asterix, and again, IMHO that is something > that > *should* require additional thought and oversight. As for build and > test, > my feeling is that it will make it considerably harder, or at the very > least slower, simply due to doubling the Maven overhead. > > I do not feel that merging the projects to either fit in better with > Apache, or to game the Apache popularity indexes, is a good trade-off. > > Ceej > aka Chris Hillery > > On Mon, Jun 1, 2015 at 12:02 PM, Yingyi Bu wrote: > >> Hi folks, >> >> Should we merge hyracks, asterixdb, and potentially pregelix/imru >> into >> the same repository? It will make build, fix, and code review >> process >> much simpler. >> An example is that everything built on top of Spark lives in the >> same >> repository: https://github.com/apache/spark. That's also why Spark >> is >> the most active Apache project now, due to its commit frequency. >> Does anyone have concerns for merging the hyracks and asterixdb >> repositories? >> Thanks! >> >> Best, >> Yingyi >> >> >> On Wed, Apr 22, 2015 at 10:13 PM, Till Westmann >> wrote: >> >>> Ok, let’s find out what is the “more work” part before we >>> decide :) >>> >>> We should already have the SGA (as it’s part of the SGA that Mike >>> sent >>> in) and it seemed to me that all we’re need to do “later” >>> (e.g. next >>> week/month) would be to >>> a) vote on bringing it into AsterixDB (that would be an incubator >>> vote I >>> assume) and >>> b) asking infra for another git repository. >>> So the extra work would be the vote on the incubator list. >>> Is that right or is there something else we’d need to do? >>> >>> Cheers, >>> Till >>> >>> On Apr 22, 2015, at 10:04 PM, Mattmann, Chris A (3980) < >>> chris.a.mattmann@jpl.nasa.gov> wrote: >>> >>> Hey Mike and team, >>> >>> Thanks for bringing this to the list. I think these are precisely >>> the type of conversations that we want to have here at the ASF and >>> as part of our Incubating project. Having these discussions in the >>> community here at the ASF (which is now the Apache AsterixDB >>> community) >>> is great. >>> >>> My opinion - it’s fine either way. I’m happy if you guys want to >>> bring Pregelix into the code base here via AsterixDB. It’s easily >>> reversible and incremental. If you want to spin out Pregelix later >>> as its own TLP and it’s shown to have its own community we can >>> file a board resolution to do that. Heck, nothing stops us from >>> graduating 2 Incubator projects=>TLPs out of this effort even in >>> the Incubator. That’s fine. If you want to wait and bring it in >>> later, it will definitely be more work - so let’s call a spade a >>> spade there. But if you want to do that that’s fine too. >>> >>> My personal recommendation - bring it in - won’t hurt and we can >>> always pivot in the ways above later. >>> >>> Cheers, >>> Chris >>> >>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Chris Mattmann, Ph.D. >>> Chief Architect >>> Instrument Software and Science Data Systems Section (398) >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>> Office: 168-519, Mailstop: 168-527 >>> Email: chris.a.mattmann@nasa.gov >>> WWW: http://sunset.usc.edu/~mattmann/ >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Adjunct Associate Professor, Computer Science Department >>> University of Southern California, Los Angeles, CA 90089 USA >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> >>> >>> >>> >>> >>> -----Original Message----- >>> From: Michael Carey >>> Date: Tuesday, April 21, 2015 at 11:49 AM >>> To: Chris Mattmann , Till Westmann >>> >>> Cc: Chris Hillery , Ian Maxon , >>> Yingyi >>> Bu , "dev@asterixdb.incubator.apache.org" >>> >>> Subject: Re: Migration of git repository >>> >>> Sure! Let me clarify the issue for everyone (and broaden the >>> question). >>> >>> One of the technical by-products of the AsterixDB project is a graph >>> analytics package called Pregelix - as the name suggests, it is a >>> "knock >>> off" of Pregel, as are packages like Giraph. What's unique about >>> Pregelix is that it actually scales without OOM'ing >>> - under the covers it uses database join processing techniques. You >>> can >>> find out more about it by visiting >>> http://pregelix.ics.uci.edu/ and/or by skimming the attached paper - >>> check out the experimental results compared to other popular >>> alternatives. Anyway, we have made it freely available (as we do >>> all of >>> our AsterixDB-related >>> research products) and we were thinking that we should simply >>> include it >>> under the AsterixDB project - kind of like Spark has subprojects for >>> SQL, >>> streams, graphs, etc. As a result, I listed it on the list of >>> transferred artifacts when I sent in the licensing >>> form the other day. (So we at least have that step done.) Its code >>> conntributors have been a small subset of the AsterixDB team; it was >>> a >>> small sub-project, basically. (Mostly just Yingyi Bu!) >>> >>> Pregelix is kind of a sibling of Apache VXQuery in that its runtime >>> is >>> based on Hyracks but it hasn't otherwise been AsterixDB-dependent. >>> However, we have just finished teaching it to read/write directly >>> from >>> AsterixDB native storage - instead of just HDFS >>> - so now it has an AsterixDB dependency, and we are using it as a >>> driving example of how to couple AsterixDB to other analytic >>> engines. >>> >>> Rather than going through another exercise to open-source this >>> separately, it seemed like we could take this approach. >>> >>> Thoughts? >>> Cheers, >>> Mike >>> >>> >>> On 4/21/15 7:45 AM, Mattmann, Chris A (3980) wrote: >>> >>> >>> Yes, in fact, this whole conversations should be happening on >>> the dev list. OK for me to CC them on my reply? >>> >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Chris Mattmann, Ph.D. >>> Chief Architect >>> Instrument Software and Science Data Systems Section (398) >>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >>> Office: 168-519, Mailstop: 168-527 >>> Email: chris.a.mattmann@nasa.gov >>> WWW: http://sunset.usc.edu/~mattmann/ >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> Adjunct Associate Professor, Computer Science Department >>> University of Southern California, Los Angeles, CA 90089 USA >>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >>> >>> >>> >>> >>> >>> >>> -----Original Message----- >>> From: "Michael J. Carey" >>> > >>> Date: Tuesday, April 21, 2015 at 3:13 AM >>> To: Till Westmann >> > >>> Cc: Chris Hillery >> >, Ian >>> Maxon >, >>> Yingyi >>> Bu >> >, >>> Chris Mattmann >>> >>> >> > >>> Subject: Re: Migration of git repository >>> >>> + Yingyi on the Pregelix Q. Should we also ask Chris M for advice >>> on >>> that? >>> On Apr 20, 2015 4:23 PM, "Till Westmann" >>> > wrote: >>> >>> Hi Ian, >>> >>> >>> That’s a good question - and I don’t know the answer. >>> We’ve got 2 repos so far: >>> https://issues.apache.org/jira/browse/INFRA-9212https://issues.apache.org/ >>> jira/browse/INFRA-9306 >>> so we should have space for Hyracks and AsterixDB. >>> >>> >>> I think that there’s an open questions about Pregelix, but maybe >>> that >>> shouldn’t keep us from going ahead. >>> >>> >>> I further think that it would be great if you could send an e-mail >>> to >>> dev@asterixdb.incubator.apache.org< >>> mailto:dev@asterixdb.incubator.apache.o >>> >>> rg> >> > and ask if it’s ok to >>> import >>> our git repo(s) or if something else needs to be done first. (I >>> could >>> send that e-mail as well, but it would be great if there were more >>> non-Till e0mails on the list :) ) >>> >>> >>> Cheers, >>> Till >>> >>> >>> On Apr 20, 2015, at 4:07 PM, Ian Maxon >>> > wrote: >>> >>> Hi Mike, Chris and Till, >>> >>> >>> Since (I think?) the paperwork for the software grant is done now, >>> should >>> I copy our GC branches over to the ASF git repositories now ( as >>> well as >>> making it a mirror in the Gerrit commit hook script)? >>> >>> >>> Thanks, >>> - Ian >>> >>> >>> >>