Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <0D278447-6957-4A84-9CFE-9D0905AAF72C@cloudera.com>
References: <CADZba-Sxdcuatzd2PwSG5BZNJuMHwpauc=D1+XFKGZs+kFGpFw@mail.gmail.com>
 <CAD5746=+=2p8xSbHUH2wV2s1VjGQTJ-GoXHo5Tg7TFn9xfXo2w@mail.gmail.com>
 <CADZba-RXGgz5Wio1dfJy_vuxdKjzm8rqycv=HuNJs0cWpxfSpQ@mail.gmail.com> <0D278447-6957-4A84-9CFE-9D0905AAF72C@cloudera.com>
From: Jim Apple <jbapple@cloudera.com>
Date: Thu, 26 May 2016 08:32:28 -0700
Message-ID: <CAC-pSX05AYmuKduM7NZ_ds-y_X3Nh=MeOmjidObF0p2Rx-tjSA@mail.gmail.com>
Subject: Re: IMPALA-3223
To: Michael Ho <kwho@cloudera.com>
Cc: Henry Robinson <henry@apache.org>, Daniel Hecht <dhecht@cloudera.com>,
	impala-cr <impala-cr@cloudera.com>, dev@impala.incubator.apache.org
Content-Type: text/plain; charset=UTF-8
archived-at: Thu, 26 May 2016 15:33:17 -0000

I agree. I also think that it's a shame Impala depends so precariously
on the exact versions of various packages. It is especially concerning
to me when those are CDH packages, or when those packages are
maintained by cloudera on cloudera's github account.

On Wed, May 25, 2016 at 5:22 PM, Michael Ho <kwho@cloudera.com> wrote:
> Other than the logistics reasons Tim mentioned, my thinking is that
> thirdparty and toolchain overlap a lot so it would be easier to have just
> one location for everything.
>
> Thanks,
> Michael
>
> On May 25, 2016, at 16:46, Henry Robinson <henry@apache.org> wrote:
>
> On 25 May 2016 at 16:33, Michael Ho <kwho@cloudera.com> wrote:
>>
>> Yes. The biggest dependency in thirdparty now is the reliance of the CDH
>> components, namely
>> hadoop, hbase, hive, sentry and some llama stuff. The short term goal as
>> discussed in another email
>> thread is to initially put a publicly available released version of those
>> CDH components in S3 so a user
>> can get all the components they need to build and run tests by just
>> running the script to bootstrap the
>> toolchain. Internally, we can replace those CDH components with our own
>> builds of those components
>> by somehow instructing the toolchain boostrapping script to pull from our
>> internal repo if needed. The
>> users can in theory specify their own repo for those components too.
>> Whether we will converge to
>> the Apache version of things in the long run is not the goal of this bug.
>>
>> The majority of the stuff left in thirdparty are mostly duplicates of what
>> exist already in
>> the toolchain directory: avro-c, sasl, gflags, glog, gperftools, gtest,
>> lz4, openldap, rapidjson, re2, snappy,
>> thrift so we may as well remove them.
>>
>> After all the changes above, thirdparty will be pretty much empty except
>> for mustache and squeasel.
>> If I recalled correctly from the meeting we had a month ago before you
>> went on leave, you pointed out
>> that they can reside in util directory as they are actually compiled
>> together with the webserver.cc. I am
>> also open to keeping them in thirdparty directory as this means we can now
>> have a thirdparty directory
>> in the ASF repo upstream after all the trimming above.
>
>
> Thanks for the explanation. I still don't see from your response why you
> think Squeasel and Mustache should move to util/, even if, as we discussed,
> they can (there are good reasons to consider moving them - maybe we don't
> want thirdparty/ any more, maybe we don't want to treat external compile
> dependencies any differently).
>
> The argument I can see in favour of not moving them is that any external
> compile-time dependencies are in an easy-to-find place, which probably makes
> getting an Apache release together just a little easier since we can split
> the code into "developed as part of Impala" and "developed externally".
>
> I'm not going to stand in the way of a small change like this, so feel free
> to GVM if you think best.
>
>>
>>
>> Michael
>>
>> On Wed, May 25, 2016 at 3:30 PM, Henry Robinson <henry@apache.org> wrote:
>>>
>>> I thought the long-term goal was to remove any heavyweight dependencies
>>> that are checked into the repository. It's not the existence of thirdparty/
>>> that's the problem there, it's what we were keeping in it.
>>>
>>> Mustache and Squeasel change very infrequently, and are only a few K in
>>> total. Do you think we should remove thirdparty/ for other reasons?
>>>
>>> On 25 May 2016 at 14:20, Michael Ho <kwho@cloudera.com> wrote:
>>>>
>>>> The long term goal is to remove thirdparty. I thought we discussed about
>>>> it.
>>>>
>>>> On Wed, May 25, 2016 at 2:13 PM, Henry Robinson <henry@cloudera.com>
>>>> wrote:
>>>>>
>>>>> (Sorry, not near Gerrit) why not keep them in thirdparty/? That makes
>>>>> their provenance clear.
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>> > On May 25, 2016, at 2:06 PM, Dan Hecht (Code Review)
>>>>> > <gerrit@cloudera.org> wrote:
>>>>> >
>>>>> > Dan Hecht has posted comments on this change.
>>>>> >
>>>>> > Change subject: IMPALA-3223: Move squeasel and mustache to the util
>>>>> > directory
>>>>> >
>>>>> > ......................................................................
>>>>> >
>>>>> >
>>>>> > Patch Set 1: Code-Review+2
>>>>> >
>>>>> > Okay.
>>>>> >
>>>>> > --
>>>>> > To view, visit http://gerrit.cloudera.org:8080/3206
>>>>> > To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
>>>>> >
>>>>> > Gerrit-MessageType: comment
>>>>> > Gerrit-Change-Id: I782e493bc28086a1587274b3c474ea6b6f201855
>>>>> > Gerrit-PatchSet: 1
>>>>> > Gerrit-Project: Impala
>>>>> > Gerrit-Branch: cdh5-trunk
>>>>> > Gerrit-Owner: Michael Ho <kwho@cloudera.com>
>>>>> > Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
>>>>> > Gerrit-Reviewer: Jim Apple <jbapple@cloudera.com>
>>>>> > Gerrit-Reviewer: Michael Ho <kwho@cloudera.com>
>>>>> > Gerrit-HasComments: No
>>>>> >
>>>>> > --
>>>>> > You received this message because you are subscribed to the Google
>>>>> > Groups "impala-cr" group.
>>>>> > To unsubscribe from this group and stop receiving emails from it,
>>>>> > send an email to impala-cr+unsubscribe@cloudera.com.
>>>>> > For more options, visit
>>>>> > https://groups.google.com/a/cloudera.com/d/optout.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Michael
>>>
>>>
>>
>>
>>
>> --
>> Thanks,
>> Michael
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "impala-cr" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to impala-cr+unsubscribe@cloudera.com.
>> For more options, visit https://groups.google.com/a/cloudera.com/d/optout.
>
>