Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
Received-SPF: pass (nike.apache.org: domain of jsichi@fb.com designates
 66.220.144.146 as permitted sender)
From: John Sichi <jsichi@fb.com>
To: "<user@hive.apache.org>" <user@hive.apache.org>
CC: "<user@pig.apache.org>" <user@pig.apache.org>
Subject: Re: [VOTE] Sponsoring Howl as an Apache Incubator project
Thread-Topic: [VOTE] Sponsoring Howl as an Apache Incubator project
Thread-Index: AQHLw8GGh3oqa5NstEqTQKjsK6uDB5PwhDcAgAAuLoA=
Date: Thu, 3 Feb 2011 19:38:16 +0000
Message-ID: <AAE2BD94-9CC9-44FA-B793-09DCA3C46CB7@fb.com>
References: <84261DF7-6B37-4F5F-B10C-056E8E9CE4E6@yahoo-inc.com>
 <AANLkTinJuWcip66+hN9BmLcXWeab9LD19unFZVs-qoD7@mail.gmail.com>
 <AANLkTink+qAjxjf=ZZ5Y4U8Dsc-iVOQwmRy=dZM26iJC@mail.gmail.com>
In-Reply-To: <AANLkTink+qAjxjf=ZZ5Y4U8Dsc-iVOQwmRy=dZM26iJC@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <D7854A5471155E4FA3A204C2A78B071C@fb.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Besides the fact that the refactoring required is significant, I don't thin=
k this is possible to do quickly since:

1) Hive (unlike Pig) requires a metastore

2) Hive releases can't depend on an incubator project

It's worth pointing out that Howl is already using Hive's CLI+DDL (not just=
 metastore).  That's a huge amount of code.  In biological terms, Howl has =
the same DNA as Hive (plus some new Howl-specific genes on a separate plugi=
n chromosome), but only a subset of the Hive genes are expressed when runni=
ng Howl; the rest are just junk DNA from Howl's perspective.

It's not clear yet that refactoring is worth the effort even in the end sta=
te.  We can achieve the desired compatibility by keeping the current approa=
ch but removing the Hive code copy from Howl, instead creating a dependency=
 from Howl to Hive.  In this case, graduating to become a Hive subproject m=
ight be the correct exit from the incubator.

If we do go ahead with pulling the metastore out of Hive, it might make mos=
t sense for Howl to become its own TLP rather than a subproject.

In the incubator proposal, we have mentioned these issues, but we've attemp=
ted to avoid prejudicing any decision.  Instead, we'd like to assess the pr=
os and cons (including effort required and impact expected) for both approa=
ches as part of the incubation process.

I don't have any voting rights on Pig but obviously I'm +1 on the proposal =
for incubation.

JVS

On Feb 3, 2011, at 8:52 AM, Jay Booth wrote:

> Food for thought, what if the metastore were moved to Howl more
> aggressively?  It seems like the end state everyone's aiming for is
> that Hive and Pig share Howl as a metastore layer, which makes all
> kinds of sense..  would it increase the chances of long term success
> if you guys just went for it and introduced the Hive->Howl dependency
> as soon as possible?  It would probably create some short term
> disruption but it could be more healthy for Howl assuming that things
> were worked out, design choices could be validated faster, you have
> that end-to-end "it works" thing going, etc.
>=20
> On Thu, Feb 3, 2011 at 11:43 AM, Ashutosh Chauhan <hashutosh@apache.org> =
wrote:
>> +1
>>=20
>> On Wed, Feb 2, 2011 at 13:18, Alan Gates <gates@yahoo-inc.com> wrote:
>>> Howl is a table management system built to provide metadata and storage
>>> management across data processing tools in Hadoop (Pig, Hive, MapReduce=
,
>>> ...).  You can learn more details at http://wiki.apache.org/pig/Howl.  =
For
>>> the last six months the code has been hosted at github.  The Howl team =
would
>>> like to move the project into the Apache Incubator.  You can see the
>>> proposal for the project at http://wiki.apache.org/incubator/HowlPropos=
al.
>>>=20
>>> In order to be accepted as an Incubator project Howl needs a Sponsoring
>>> project.  I propose that we, the Pig project, sponsor Howl.  By sponsor=
ing
>>> Howl we are saying that we believe it is a good fit for the ASF and tha=
t we
>>> will assist the Howl project to succeed.  You can read full details of
>>> sponsoring a project at
>>> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#=
Sponsor.
>>>=20
>>> Our bylaws don't explicitly cover such a vote, but I think lazy majorit=
y
>>> should be reasonable.  All votes are welcome, PMC member votes will be
>>> binding.
>>>=20
>>> Clearly I'm +1.
>>>=20
>>> Alan.
>>>=20
>>=20