Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: <f590649e-a83c-9db6-da5c-6cfb715a33c1@apache.org>
References: <d849cc26-3cb8-2eb4-2bfb-702d44eaafb9@apache.org>
 <CAL5zq9aQ-UekUt==_XcWvq=nCKP-+9DKaxPxJLgt=hD4KSH2iA@mail.gmail.com>
 <06fc51f8-814a-2899-c62b-15c7f1aeca0f@apache.org> <b99fcfae-b526-9c38-8102-b2a528f8d28f@apache.org>
 <CAL5zq9Y4bjF5HyB_WsFDQEoAtD4j+pF4TZe+0f-RcBFrWTpf3A@mail.gmail.com>
 <0174acbf-478d-eeee-32d5-6fb5255ad7d2@apache.org> <007301d36d57$7b7abce0$727036a0$@gmail.com>
 <CAGUtCHr0sXU7PoKfyjM8Dc5xE+x6s-j4DX-dN1cpVt5NftQubA@mail.gmail.com>
 <3d2e197b-f3ad-4c37-69a1-b612f9a03a28@apache.org> <CAGUtCHpcxiAMYh_DSLXCpDOEYDa6iDrrTTY5YKPnaOVKwC6RzA@mail.gmail.com>
 <e2b89fd9-2100-cb66-69f1-8fe93998aa6f@apache.org> <CAGUtCHrdsd7kZJCJw2VM2gRHCTgT3vp6mfaL_LtNgTKT3S5OKw@mail.gmail.com>
 <f3a680b6-715a-3e27-832f-0ddffc175b20@apache.org> <CAGUtCHrLffF34LUdGWoaRWp2RjpBYjibShNWfjr2mxFSJRcROg@mail.gmail.com>
 <dba63217-3652-9e4d-d38f-32cb2b7e1eee@apache.org> <CAL5zq9ZddzLT8yBkH8VALp7WfPARDyY=QM7+P=QHacGViT6jcQ@mail.gmail.com>
 <f590649e-a83c-9db6-da5c-6cfb715a33c1@apache.org>
From: Keith Turner <keith@deenlo.com>
Date: Wed, 6 Dec 2017 11:29:23 -0500
Message-ID: <CAGUtCHp9er5gMmoBmqFDnG07gh7oShbMNdw6X4BONAbTf2ngSA@mail.gmail.com>
Subject: Re: [DISCUSS] Hadoop3 support target?
To: Accumulo Dev List <dev@accumulo.apache.org>
Content-Type: text/plain; charset="UTF-8"
archived-at: Wed, 06 Dec 2017 16:29:36 -0000

On Wed, Dec 6, 2017 at 11:28 AM, Josh Elser <elserj@apache.org> wrote:
> 1.a sounds better to me.

why?

>
> A would be the ideal solution, I think B is the next best if A doesn't work.
>
> I need to get the Hadoop3 compatibility fixed, so I'll be investigating the
> Hadoop shaded artifacts this week.
>
>
> On 12/5/17 6:43 PM, Christopher wrote:
>>
>> I was wondering about Hadoop 3 shading and whether that would help us. It
>> would be really nice if it could, or if there was some other class path
>> solution that was easy.
>>
>> I think there are two major issues in this thread. The first is the API
>> problems. The second is the Hadoop 3 support. They are related, but I
>> think
>> quickly dealing with the API issues can clarify what our options are for
>> Hadoop 3.
>>
>>
>>
>>
>> To fix the API, I would like to get consensus on proceeding with this
>> path:
>>
>> 1. Rename 1.8.2-SNAPSHOT to 1.9.0-SNAPSHOT and deprecate the existing
>> ZooKeeperInstance constructor which takes a Configuration
>>      a) Deprecate ClientConfiguration and replace with ClientConfig (or a
>> better name) which does not extend Configuration or have API leak
>> problems,
>> and add a new ZKI constructor for this
>>      b) Ignore extends for now, and drop it from ClientConfiguration in
>> 2.0
>> with a break (can't deprecate superclass), and add new ZKI constructor for
>> more specific ClientConfiguration next to deprecated one
>> 2. Drop deprecated stuff from 2.0 branch (and extends, if option 1.b was
>> chosen)
>> 3. Plan a 1.9.0 release instead of 1.8.2
>>
>> I prefer 1.a over 1.b, personally, but I've been tossing back and forth. I
>> would need input on which is best. There are pros and cons to both,
>> regarding churn, and source and binary compatibility.
>>
>>
>>
>>
>> Once we deal with the API, our options for Hadoop 3 become:
>>
>> A. Use Hadoop 3 shaded artifacts or some other class path solution (such
>> as
>> getting lucky identifying a version of commons-beanutils that works for
>> both)
>> B. Shade in 1.9 with a breaking change
>> C. Create a 1.9 version named 2.0, so we can do a breaking change without
>> semver violation; shade in this version
>> D. Shade in the branch we're currently calling 2.0
>>
>> I think we can defer that decision pending some further
>> investigation/experimentation into what works, and deal with it after
>> dealing with steps 1-3 above (but soon after, hopefully).
>>
>>
>>
>> On Tue, Dec 5, 2017 at 3:58 PM Josh Elser <elserj@apache.org> wrote:
>>
>>> Another potential suggestion I forgot about: we try to just move to the
>>> Hadoop shaded artifacts. This would invalidate the need to do more, but
>>> I have no idea how "battle-tested" those artifacts are.
>>>
>>> On 12/5/17 3:52 PM, Keith Turner wrote:
>>>>
>>>> If we do the following.
>>>>
>>>>    * Drop ZooKeeperInstance.ZooKeeperInstance(Configuration config)
>>>
>>> method.
>>>>
>>>>    * Drop extends from ClientConfig
>>>>    * Add a method ZooKeeperInstance.ZooKeeperInstance(ClientConfig
>>>> config)
>>>>
>>>> Then this will not be binary compatible, so it will still be painful
>>>> in many cases.   It may be source compatible.
>>>>
>>>> For example the following will be source (but not binary) compatible.
>>>>
>>>>     ClientConfiguration cc = new ClientConfiguration(file);
>>>>     //when compiled against older version of Accumulo will bind to
>>>> method with commons config signature
>>>>     //when recompiled will bind to clientconfig version of method
>>>>     ZooKeeperInstance zki = new ZooKeeperInstance(cc);
>>>>
>>>> The following would not be source or binary compatible.
>>>>
>>>>     Configuration cc = new ClientConfiguration(file);
>>>>     ZooKeeperInstance zki = new ZooKeeperInstance(cc);
>>>>
>>>>
>>>> On Tue, Dec 5, 2017 at 3:40 PM, Josh Elser <elserj@apache.org> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 12/5/17 3:28 PM, Keith Turner wrote:
>>>>>>
>>>>>>
>>>>>> On Tue, Dec 5, 2017 at 2:53 PM, Josh Elser<elserj@apache.org>  wrote:
>>>>>>>
>>>>>>>
>>>>>>> Interesting. What makes you want to deprecate ClientConfig entirely?
>>>>>>>
>>>>>>> I'd be worried about removing without sufficient thought of
>>>
>>> replacement
>>>>>>>
>>>>>>> around. It would be a bit "churn-y" to introduce yet another way that
>>>>>>> clients have to connect (since it was introduced in 1.6-ish?).
>>>>>>> Working
>>>>>>> around the ClientConfig changes was irritating for the downstream
>>>>>>> integrations (Hive, most notably).
>>>>>>
>>>>>>
>>>>>> Ok maybe thats a bad idea, not looking to cause pain.  Here were some
>>>>>> of my goals.
>>>>>>
>>>>>>     * Remove commons config from API completely via deprecation cycle.
>>>>>>     * Introduce API that supports putting all props needed to connect
>>>>>> to
>>>>>> Accumulo in an API.
>>>>>>
>>>>>> I suppose if we want to keep ClientConfig class in API, then there is
>>>>>> no way to remove commons config via a deprecation cycle??  We can't
>>>>>> deprecate the extension of commons config, all we can do is just drop
>>>>>> it at some point.
>>>>>>
>>>>>
>>>>> My line of thinking is that the majority of the time, we're creating a
>>>>> ClientConfiguration by one of:
>>>>>
>>>>> * ClientConfiguration#loadDefault()
>>>>> * new ClientConfiguration(String)
>>>>> * new ClientConfiguration(File)
>>>>>
>>>>> Granted, we also inherit/expose a few other things (notably extending
>>>>> CompositeConfiguration and throwing ConfigurationException). I would be
>>>>> comfortable with dropping those w/o deprecation. I have not seen
>>>
>>> evidence
>>>>>
>>>>> from anyone that they are widely in use by folks (although I've not
>>>>> explicitly asked, either).
>>>
>>>
>>
>