Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANcXBFMsEvLjtXsP5YUHUyobn+tgxiJHLBTTHm8Aqj7fn_0a=Q@mail.gmail.com>
References: <0454CD1B-9018-4D5F-A376-CD29525FF87B@apache.org>
	<479655474.308681.1458236651551.JavaMail.yahoo@mail.yahoo.com>
	<051E7389-1779-44A8-8BEF-DA440305F9D5@apache.org>
	<CANLc_9KG9T_XGSDVy99BLdqFbh-HC5kgfaMfuu9O_QCZhiUAhw@mail.gmail.com>
	<CANcXBFPgoe_vvAJFHwKkO7J7NAmLOTDjUSZ4-a7LbFxNf6mg7g@mail.gmail.com>
	<CANLc_9+L7EF+F+GTA5qu9rGqKoXtBYFi=u-bg4sD3W-s4MxmBQ@mail.gmail.com>
	<CANcXBFM2nrLVEbTXoqTjKQ1vCs2oEcYSq+5WpRYFa30PSHLLaA@mail.gmail.com>
	<CANLc_9KnvDgmKHF02f_s0qsB8j=GrC5sqZkwW1U__VtoFaaxiw@mail.gmail.com>
	<CANcXBFPzD5rgkuqPvH0sZ_T33J7L7s0VPuhv_whXve8ssWgUjA@mail.gmail.com>
	<CANLc_9++sRKWp2pzJzeOLMi9ewfM2UGsax2dMkF-bpcXkvQbXA@mail.gmail.com>
	<CANcXBFPSbczPnTZryCo51onVbeqQgsaR0ec1aPnJmD5m2nSGUw@mail.gmail.com>
	<CAA+BczRetxe-+M_FwL-qWw1euwTbZKfYbWr9BRZfxhGfeRa_gA@mail.gmail.com>
	<CANcXBFOWP5rRYGW8cTztsgd3kxX-OosehjzpF9QKYjROGi4kRA@mail.gmail.com>
	<58E62111-0064-4291-B376-D40FFCFFF2A9@apache.org>
	<CANcXBFOqJv_hrCaaokYJ8ccfjO6SDzujFNwsvH7YkQCtXe-62w@mail.gmail.com>
	<CAA+BczTxzN+pYDqNuEEe50HN1_758E=wciJ99rdYsGqi+za8Ew@mail.gmail.com>
	<CANcXBFMsEvLjtXsP5YUHUyobn+tgxiJHLBTTHm8Aqj7fn_0a=Q@mail.gmail.com>
Date: Fri, 1 Apr 2016 12:48:06 -0400
Message-ID: 
 <CAA+BczS8wxQ511ws+FL3SE7rk_5xUsHU_t1RabnuE1qyNi1RDw@mail.gmail.com>
Subject: Re: Zookeeper with SSL release date
From: Jason Rosenberg <jbr@squareup.com>
To: user@zookeeper.apache.org
Content-Type: multipart/alternative; boundary=089e0115f8c6aa7031052f6f248a

--089e0115f8c6aa7031052f6f248a
Content-Type: text/plain; charset=UTF-8

sure sounds like a bad policy not to allow reconfig without ACL's, but they
are essentially unrelated functionality, nonetheless....

On Fri, Apr 1, 2016 at 12:18 PM, Alexander Shraer <shralex@gmail.com> wrote:

> Because using reconfig without ACLs any client can remove the servers (or
> replace them with a different set of servers
> or change their configuration parameters) and break the system.
>
> On Fri, Apr 1, 2016 at 8:59 AM, Jason Rosenberg <jbr@squareup.com> wrote:
>
> > I think these orthogonal concerns.  Why limit reconfig to ACL users only?
> >
> > On Thu, Mar 31, 2016 at 11:37 PM, Alexander Shraer <shralex@gmail.com>
> > wrote:
> >
> > > Citing Patrick:
> > >
> > > > If you're running zk w/o security turned on and suddenly folks can do
> > > reconfig
> > > > operations it's going to potentially be a problem.
> > > ...
> > > > Rather than force people to turn on kerberos (etc...) we could
> instead
> > > > have the feature off
> > >
> > > From this I understood that the concern is mostly about users that
> DON'T
> > > use ACLs. My proposal is to disable
> > > reconfig/getconfig for all such users, forcing users who want reconfig
> to
> > > also use ACLs. Users who do use ACLs
> > > don't have to use reconfig and will have to set the ACLs on the config
> > > znode before they can use it.
> > >
> > > In preprequestprocessor where acls are checked for reconfig operation
> we
> > > can check that:
> > >
> > > skipACL = false && nodeRecord.acl != null && nodeRecord.acl.size() != 0
> > >
> > > meaning you're using ACLs, and have actually set ACLs on the config
> node.
> > >
> > > For getConfig its a bit trickier since its just a getData on the server
> > > side (for efficiency
> > > of reads, we avoided checking whether path == config znode). What we
> > could
> > > do is before sending
> > > the operation to the server check skipACL = false and maybe also issue
> a
> > > getACL call to check that
> > > nodeRecord.acl != null && nodeRecord.acl.size() != 0
> > > and only then issue a getData. This part is not air tight but its
> > probably
> > > sufficient.
> > >
> > > And of course we can emphasize the need for ACLs on this znode in the
> > > release.
> > >
> > >
> > > On Thu, Mar 31, 2016 at 1:11 PM, Flavio Junqueira <fpj@apache.org>
> > wrote:
> > >
> > > > I think Jason is saying that this is orthogonal in the following
> sense.
> > > > You set ACLs because you care about authentication/authorization in
> > your
> > > > cluster, but you may not want reconfig enabled, it just happened that
> > you
> > > > wanted to use ACLs.
> > > >
> > > > Perhaps you can elaborate a bit on how you think we can perform this
> > ACL
> > > > check? What would you check precisely?
> > > >
> > > > -Flavio
> > > >
> > > > > On 24 Mar 2016, at 21:19, Alexander Shraer <shralex@gmail.com>
> > wrote:
> > > > >
> > > > > I'm not so sure its orthogonal. The question is whether someone
> would
> > > > ever
> > > > > want to use reconfig without ACLs,
> > > > > as this allows any client to reconfigure the servers away or add a
> > > bunch
> > > > of
> > > > > servers that shouldn't be there :) and whether we should facilitate
> > > this
> > > > > knowing its insecure.
> > > > >
> > > > > Requiring ACLs solves the security concern for both reconfig and
> > > > getconfig.
> > > > > For example, if you don't want your clients to know the list of
> > > servers,
> > > > > limit their read permissions on the configuration znode.
> > > > >
> > > > > On Thu, Mar 24, 2016 at 2:11 PM, Jason Rosenberg <jbr@squareup.com
> >
> > > > wrote:
> > > > >
> > > > >> seems like an orthogonal requirement?
> > > > >>
> > > > >> On Thu, Mar 24, 2016 at 3:37 PM, Alexander Shraer <
> > shralex@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> How about a simpler alternative to the proposed flag for
> reconfig:
> > a
> > > > >> check
> > > > >>> in the code that requires ACLs to be set.
> > > > >>> If people want to use reconfig, they should use ACLs too.
> > > > >>>
> > > > >>> What do you think ?
> > > > >>>
> > > > >>> Alex
> > > > >>>
> > > > >>> On Mon, Mar 21, 2016 at 9:58 PM, Patrick Hunt <phunt@apache.org>
> > > > wrote:
> > > > >>>
> > > > >>>> I would say if in doubt add a safety. (a config parameter to
> turn
> > it
> > > > >>>> off). Cost is almost zero and worst case it will just give us
> > peace
> > > of
> > > > >>>> mind. ;-)
> > > > >>>>
> > > > >>>> Patrick
> > > > >>>>
> > > > >>>> On Mon, Mar 21, 2016 at 9:41 PM, Alexander Shraer <
> > > shralex@gmail.com>
> > > > >>>> wrote:
> > > > >>>>> ok, thanks for the suggestion, I'll look into it. For reconfig
> I
> > > > >> think
> > > > >>>> its
> > > > >>>>> pretty clear that its an admin
> > > > >>>>> functionality. I just always imagined that its controlled via
> > acls,
> > > > >>> but I
> > > > >>>>> understand
> > > > >>>>> the concerns now.
> > > > >>>>>
> > > > >>>>> getConfig returns the dynamic config (list of all servers with
> > all
> > > > >>> ports
> > > > >>>>> and quorum system if defined)
> > > > >>>>> and has an option to filter that info and just return the
> server
> > > > >>>> connection
> > > > >>>>> string (server and client port only).
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Mon, Mar 21, 2016 at 9:32 PM, Patrick Hunt <
> phunt@apache.org>
> > > > >>> wrote:
> > > > >>>>>
> > > > >>>>>> On Mon, Mar 21, 2016 at 9:14 PM, Alexander Shraer <
> > > > >> shralex@gmail.com>
> > > > >>>>>> wrote:
> > > > >>>>>>> I don't think that getConfig should be an admin
> functionality.
> > It
> > > > >> is
> > > > >>>>>>> essential for client-side re-balancing
> > > > >>>>>>> that we implemented (all clients shoudl be able to detect
> > > > >>>> configuration
> > > > >>>>>>> changes via getConfig). It could
> > > > >>>>>>> be hidden somewhat by defining higher-level re-balancing
> > > > >>>>>>> policies (ZOOKEEPER-2016)
> > > > >>>>>>> but there hasn't been enough progress on that. Perhaps
> instead
> > > > >>>> getConfig
> > > > >>>>>>> should be controlled
> > > > >>>>>>> by a separate flag ?
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>> I believe that the Hadoop community has something we could
> use:
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/InterfaceClassification.html
> > > > >>>>>> (whether through annotations or just documenting it in the API
> > > > >>> javadoc)
> > > > >>>>>>
> > > > >>>>>> e.g. we could list getConfig as public/unstable for example
> and
> > > > >> still
> > > > >>>>>> ship it as GA. That would mark it as something that could
> change
> > > re
> > > > >>>>>> API policy.
> > > > >>>>>>
> > > > >>>>>> Is the entire config exposed through getConfig? If so then we
> > > might
> > > > >>>>>> want to enable/disable it with a flag similar to reconfig.
> Might
> > > be
> > > > >>>>>> safer to just do that if we're not sure.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> Re classification - we could do the same thing with reconfig,
> > but
> > > I
> > > > >>>>>> think that would be a mistake. If we feel strongly where it
> > should
> > > > >>>>>> live long term we should just move it now.
> > > > >>>>>>
> > > > >>>>>> Patrick
> > > > >>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On Mon, Mar 21, 2016 at 9:04 PM, Patrick Hunt <
> > phunt@apache.org>
> > > > >>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> On Mon, Mar 21, 2016 at 8:52 PM, Alexander Shraer <
> > > > >>> shralex@gmail.com
> > > > >>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>> Hi Patrick, Flavio,
> > > > >>>>>>>>>
> > > > >>>>>>>>> Since there seems to be consensus on this, I can add this
> > flag,
> > > > >>>> unless
> > > > >>>>>>>>> someone else wants to. I assume that getConfig should still
> > > > >> work
> > > > >>>>>>>> regardless
> > > > >>>>>>>>> of the flag ? is there a security concern with clients
> > knowing
> > > > >>> the
> > > > >>>>>> list
> > > > >>>>>>>> of
> > > > >>>>>>>>> servers?
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>> We've always hidden that detail from users. We don't even
> > expose
> > > > >>>> which
> > > > >>>>>>>> server you're connected to today. I remember Ben (and
> perhaps
> > > > >>>> Flavio?)
> > > > >>>>>>>> highlighting this as important to maintain although I'm not
> > > super
> > > > >>>>>>>> familiar with the specifics on why. It made sense to me
> though
> > > > >> from
> > > > >>>>>>>> the perspective that we don't want clients to make
> assumptions
> > > > >> that
> > > > >>>>>>>> probably shouldn't.
> > > > >>>>>>>>
> > > > >>>>>>>> My thinking is that we should 1) add a config option to
> enable
> > > > >>>>>>>> reconfig (off by default), 2) move reconfig specific
> > > > >> functionality
> > > > >>>>>>>> from ZooKeeper.java (including getconfig) into an "admin"
> > > > >> package,
> > > > >>>>>>>> within say a class ZooKeeperAdmin, 3) document/test use of
> > ACLs
> > > > >> for
> > > > >>>>>>>> when folks do want to enable reconfig and are also worried
> > about
> > > > >>>> auth.
> > > > >>>>>>>> (e.g. turn on kerb)
> > > > >>>>>>>>
> > > > >>>>>>>> Again, I don't see any of this as a quality issue
> personally.
> > As
> > > > >>> such
> > > > >>>>>>>> I don't see why any of this (1-3) should hold up a
> 3.5.2-alpha
> > > if
> > > > >>> we
> > > > >>>>>>>> were interested in doing such a release. Adjusting the API
> > > should
> > > > >>> be
> > > > >>>>>>>> done before we move to "beta" though. Although that seems
> > like a
> > > > >>>>>>>> pretty mechanical (eclipse/idea) type refactoring?
> > > > >>>>>>>>
> > > > >>>>>>>> Patrick
> > > > >>>>>>>>
> > > > >>>>>>>>> Cheers,
> > > > >>>>>>>>> Alex
> > > > >>>>>>>>> On Mar 21, 2016 8:34 PM, "Patrick Hunt" <phunt@apache.org>
> > > > >>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> On Thu, Mar 17, 2016 at 4:08 PM, Flavio Junqueira <
> > > > >>> fpj@apache.org
> > > > >>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>> I gotta say that I'm not super excited about this option,
> > > > >> but
> > > > >>>> for
> > > > >>>>>> some
> > > > >>>>>>>>>> reason I ended up carrying the flag. To recap, I just
> raised
> > > > >>> this
> > > > >>>>>> option
> > > > >>>>>>>>>> because it seems that there are folks interested in
> features
> > > > >> in
> > > > >>>> 3.5
> > > > >>>>>> like
> > > > >>>>>>>>>> SSL and not necessarily in reconfiguration. SSL is
> important
> > > > >> and
> > > > >>>> to
> > > > >>>>>> take
> > > > >>>>>>>>>> Kafka as an example, it sucks that we can't have a whole
> set
> > > > >> up
> > > > >>>> using
> > > > >>>>>>>> SSL.
> > > > >>>>>>>>>> For ZK, the real questions are:
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> 1- how fast can we make 3.5 stable?
> > > > >>>>>>>>>>> 2- would it be faster if we have a way of disabling
> > > > >>>>>> reconfiguration?
> > > > >>>>>>>>>>> 3- would enough users care about a stable 3.5 that has
> > > > >>>>>> reconfiguration
> > > > >>>>>>>>>> disabled?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> It is taking a long time to get 3.5 to beta. There has
> been
> > > > >>> some
> > > > >>>>>> good
> > > > >>>>>>>>>> activity around 3.5.2 release, which is a great step, but
> it
> > > > >> is
> > > > >>>>>> unclear
> > > > >>>>>>>>>> when 3.5.3 is going to come and if we will be able to make
> > 3.5
> > > > >>>> beta
> > > > >>>>>>>> then.
> > > > >>>>>>>>>> Frankly, disabling reconfiguration sounds undesirable
> > because
> > > > >> it
> > > > >>>> is
> > > > >>>>>> an
> > > > >>>>>>>>>> important feature, but I currently don't use it in
> > production,
> > > > >>> so
> > > > >>>>>> from a
> > > > >>>>>>>>>> practical point of view, I can go both ways. Whether we go
> > > > >>> through
> > > > >>>>>> the
> > > > >>>>>>>>>> trouble of doing 2 depends on users interested in that
> > option
> > > > >>> and
> > > > >>>>>> folks
> > > > >>>>>>>>>> willing to implement it.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> To answer your question, Powell, my pseudo-proposal is
> kind
> > > > >>> of a
> > > > >>>>>> funny
> > > > >>>>>>>>>> option because once the feature is stable, then we
> wouldn't
> > > > >>> need a
> > > > >>>>>>>> switch
> > > > >>>>>>>>>> any longer, so there is not need of a deprecation path, we
> > > > >> just
> > > > >>>> start
> > > > >>>>>>>>>> ignoring it from the first beta release. Until it is beta,
> > I'd
> > > > >>> say
> > > > >>>>>> that
> > > > >>>>>>>>>> default is disabled.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I would argue that we need this even when it does become
> > > > >> stable.
> > > > >>>> To
> > > > >>>>>> me
> > > > >>>>>>>>>> this is not a quality issue so much as it is an auth
> issue.
> > We
> > > > >>>> want
> > > > >>>>>> to
> > > > >>>>>>>>>> make it simple for folks to run a vanilla/old ZK cluster
> and
> > > > >> not
> > > > >>>>>> worry
> > > > >>>>>>>>>> about the security implications of having reconfig
> enabled.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Patrick
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> -Flavio
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>> On 17 Mar 2016, at 17:44, powell molleti
> > > > >>>>>> <powellm79@yahoo.com.INVALID
> > > > >>>>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Hi Flavio,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Generally do config options and command line args come
> > > > >> under
> > > > >>>> the
> > > > >>>>>> same
> > > > >>>>>>>>>> SLA as API?. I was assuming as such hence my question.
> > Perhaps
> > > > >>> if
> > > > >>>> the
> > > > >>>>>>>>>> expectation is that this config option is temporary from
> get
> > > > >> go
> > > > >>>> then
> > > > >>>>>>>> may be
> > > > >>>>>>>>>> it is ok. The default for re-config support will be
> enabled
> > or
> > > > >>>>>>>> disabled?.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I am just thinking from provisioning point of view when
> > > > >>> people
> > > > >>>>>>>> generate
> > > > >>>>>>>>>> config options etc.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>> Powell.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> On Thursday, March 17, 2016 12:12 AM, Flavio Junqueira <
> > > > >>>>>>>> fpj@apache.org>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>> Hi Powell,
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> I was thinking config file and system property server
> > side.
> > > > >>>> What's
> > > > >>>>>>>> your
> > > > >>>>>>>>>> concern with compatibility? The API itself wouldn't
> change,
> > > > >> but
> > > > >>>> the
> > > > >>>>>>>> config
> > > > >>>>>>>>>> option wouldn't exist in previous versions and would not
> > exist
> > > > >>>>>> either in
> > > > >>>>>>>>>> later stable versions of 3.5.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> -Flavio
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On 16 Mar 2016, at 22:08, powell molleti
> > > > >>>>>>>> <powellm79@yahoo.com.INVALID>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Will this option be supplied via config file/args/API?.
> > > > >> Will
> > > > >>>> this
> > > > >>>>>>>>>> option be a temporary thing i.e what about its
> > compatibility?.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> thanks
> > > > >>>>>>>>>>>>> Powell.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> On Wednesday, March 16, 2016 2:46 PM, Flavio Junqueira
> <
> > > > >>>>>>>> fpj@apache.org>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>> The main issue to sort out is stability of the API.
> There
> > > > >>> is a
> > > > >>>>>>>>>> security concern around the fact that clients can freely
> > > > >>>> reconfigure
> > > > >>>>>> the
> > > > >>>>>>>>>> ensemble. If we follow the plan that Pat proposed some
> time
> > > > >> ago:
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/zookeeper-dev/201407.mbox/%3CCANLc_9KG6-Dhm=wwfuwzNioGK70pg+iHMHPigYfJDSLf9-eq6Q@mail.gmail.com%3E
> > > > >>>>>>>>>> <
> > > > >>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://mail-archives.apache.org/mod_mbox/zookeeper-dev/201407.mbox/%3CCANLc_9KG6-Dhm=wwfuwzNioGK70pg+iHMHPigYfJDSLf9-eq6Q@mail.gmail.com%3E
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> Locking the API is the main step to move it to beta.
> > > > >> Sorting
> > > > >>>> out
> > > > >>>>>>>> bugs
> > > > >>>>>>>>>> is definitely necessary, but it isn't the main thing that
> is
> > > > >>>> keeping
> > > > >>>>>>>> 3.5 in
> > > > >>>>>>>>>> alpha.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> About making it experimental, I was raising the option
> of
> > > > >>>> having
> > > > >>>>>> a
> > > > >>>>>>>>>> switch that disables the API calls, not the code. The
> reason
> > > > >> why
> > > > >>>> that
> > > > >>>>>>>> could
> > > > >>>>>>>>>> work is that anyone using 3.5 who uses the "experimental"
> > API
> > > > >>> must
> > > > >>>>>>>> explicit
> > > > >>>>>>>>>> turn on the switch and enable the calls. If they do it,
> they
> > > > >>> need
> > > > >>>> to
> > > > >>>>>> be
> > > > >>>>>>>>>> aware that the API can change.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> I must say that I haven't really looked closely into
> > doing
> > > > >>> it,
> > > > >>>>>> and
> > > > >>>>>>>> I'm
> > > > >>>>>>>>>> not even entirely convinced that this is a good idea, but
> > > > >> since
> > > > >>>> Jason
> > > > >>>>>>>>>> raised the point, I'm exploring options.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>> -Flavio
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On 16 Mar 2016, at 20:59, Alexander Shraer <
> > > > >>>> shralex@gmail.com>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Looking at the list of ~50 blocker and critical bugs
> in
> > > > >>>>>> ZooKeeper,
> > > > >>>>>>>>>> only 3-4
> > > > >>>>>>>>>>>>>> are related to reconfig. Given this, and the fact that
> > it
> > > > >>> is
> > > > >>>>>> run in
> > > > >>>>>>>>>>>>>> production since 2012 in multiple companies, I don't
> > > > >> think
> > > > >>>> its
> > > > >>>>>> more
> > > > >>>>>>>>>>>>>> unstable than any other part of ZooKeeper.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> There are multiple reconfig-related bugs that turned
> out
> > > > >>>> really
> > > > >>>>>>>>>> difficult
> > > > >>>>>>>>>>>>>> to debug without access to the actual system or at
> least
> > > > >> to
> > > > >>>> the
> > > > >>>>>>>> Hudson
> > > > >>>>>>>>>>>>>> machines where some tests are failing. In the past,
> > Michi
> > > > >>>> and I
> > > > >>>>>>>>>> physically
> > > > >>>>>>>>>>>>>> went to Hortonworks to debug one such issue, but this
> is
> > > > >> of
> > > > >>>>>> course
> > > > >>>>>>>>>> not a
> > > > >>>>>>>>>>>>>> good method, and we weren't able to arrange such a
> visit
> > > > >>>> again.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Regarding making it optional - the reconfig logic has
> > > > >>> several
> > > > >>>>>>>>>> different
> > > > >>>>>>>>>>>>>> parts, some would be really difficult to disable
> using a
> > > > >>>>>>>> configuration
> > > > >>>>>>>>>>>>>> parameter. But the actual dynamic expansion of the
> > > > >> cluster
> > > > >>> is
> > > > >>>>>>>>>> triggered by
> > > > >>>>>>>>>>>>>> the reconfig command, so it should not affect users
> who
> > > > >>> don't
> > > > >>>>>>>> invoke
> > > > >>>>>>>>>> it.
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 1:09 PM, Flavio P JUNQUEIRA <
> > > > >>>>>>>> fpj@apache.org>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> I suppose we could give it a try. How do other folks
> > > > >> feel
> > > > >>>> about
> > > > >>>>>>>> it?
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>> -Flavio
> > > > >>>>>>>>>>>>>>> On 16 Mar 2016 19:52, "Jason Rosenberg" <
> > > > >> jbr@squareup.com
> > > > >>>>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> So, you could enable the dynamic reconfiguration
> > > > >> feature
> > > > >>>>>> behind a
> > > > >>>>>>>>>> config
> > > > >>>>>>>>>>>>>>>> option, and document that it should only be enabled
> > > > >>>>>>>> experimentally,
> > > > >>>>>>>>>> use
> > > > >>>>>>>>>>>>>>> at
> > > > >>>>>>>>>>>>>>>> your own risk.  Keep it off by default.  Allow only
> > > > >>> static
> > > > >>>>>>>> config by
> > > > >>>>>>>>>>>>>>>> default, until it's stable?
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> Jason
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 3:34 PM, Flavio Junqueira <
> > > > >>>>>>>> fpj@apache.org>
> > > > >>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Hi Jason,
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> The consumer in Kafka is pretty independent from
> the
> > > > >>> core
> > > > >>>>>>>>>> (brokers),
> > > > >>>>>>>>>>>>>>>>> that's how that project manages to make such a
> > > > >>>> separation. We
> > > > >>>>>>>> don't
> > > > >>>>>>>>>>>>>>> have
> > > > >>>>>>>>>>>>>>>>> the same with ZooKeeper as the feature we are
> talking
> > > > >>>> about
> > > > >>>>>> is
> > > > >>>>>>>>>> part of
> > > > >>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> server and the only way I see of doing what you say
> > is
> > > > >>> to
> > > > >>>>>> turn
> > > > >>>>>>>> off
> > > > >>>>>>>>>>>>>>>>> features. More specifically, we'd need to disable
> the
> > > > >>>>>> reconfig
> > > > >>>>>>>> API
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>>>>>>> do
> > > > >>>>>>>>>>>>>>>>> not allow any change to the configuration, even
> > though
> > > > >>> the
> > > > >>>>>> code
> > > > >>>>>>>> is
> > > > >>>>>>>>>>>>>>> there.
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> Reconfig here refers to the ability of changing the
> > > > >>>>>>>> configuration
> > > > >>>>>>>>>> of an
> > > > >>>>>>>>>>>>>>>>> ensemble (e.g., changing the set of servers).
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>> -Flavio
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On 16 Mar 2016, at 19:14, Jason Rosenberg <
> > > > >>>> jbr@squareup.com
> > > > >>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> So, it would seem sensible to me to have a release
> > > > >>> where
> > > > >>>> all
> > > > >>>>>>>>>> features
> > > > >>>>>>>>>>>>>>>> are
> > > > >>>>>>>>>>>>>>>>>> stable, except where noted.  E.g. mark certain
> > > > >> features
> > > > >>>> as
> > > > >>>>>> only
> > > > >>>>>>>>>>>>>>> 'alpha
> > > > >>>>>>>>>>>>>>>>>> quality', e.g. the 're-config feature'.  (I assume
> > > > >> it's
> > > > >>>>>>>> possible
> > > > >>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>> happily
> > > > >>>>>>>>>>>>>>>>>> use 3.5.X without exercising the unstable
> re-config
> > > > >>>> bits?).
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> There's precedent for doing this sort of thing in
> > > > >> other
> > > > >>>>>>>> projects,
> > > > >>>>>>>>>>>>>>> e.g.
> > > > >>>>>>>>>>>>>>>> in
> > > > >>>>>>>>>>>>>>>>>> Kafka, they've had several release where a new
> > > > >>> "Consumer
> > > > >>>>>> API"
> > > > >>>>>>>> is
> > > > >>>>>>>>>>>>>>>> shipped
> > > > >>>>>>>>>>>>>>>>>> that is available for beta-testing, but you can
> > still
> > > > >>>> just
> > > > >>>>>> use
> > > > >>>>>>>> the
> > > > >>>>>>>>>>>>>>>> older
> > > > >>>>>>>>>>>>>>>>>> stable consumer api, etc.
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> Jason
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>> On Wed, Mar 16, 2016 at 2:01 PM, powell molleti
> > > > >>>>>>>>>>>>>>>>> <powellm79@yahoo.com.invalid
> > > > >>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> Hi Doug,
> > > > >>>>>>>>>>>>>>>>>>> Is 3.5 being an alpha release preventing you from
> > > > >>> using
> > > > >>>>>> it?.
> > > > >>>>>>>> Or
> > > > >>>>>>>>>> have
> > > > >>>>>>>>>>>>>>>> you
> > > > >>>>>>>>>>>>>>>>>>> run into issues with it?. In general perhaps ZK
> 3.5
> > > > >>>> being
> > > > >>>>>>>>>> labeled as
> > > > >>>>>>>>>>>>>>>>> alpha
> > > > >>>>>>>>>>>>>>>>>>> might not be fair, since it is far more stable
> then
> > > > >>> what
> > > > >>>>>> most
> > > > >>>>>>>>>> people
> > > > >>>>>>>>>>>>>>>>>>> associate an alpha release to be.
> > > > >>>>>>>>>>>>>>>>>>> Perhaps if you do not use re-config feature may
> be
> > > > >> it
> > > > >>>> will
> > > > >>>>>>>> just
> > > > >>>>>>>>>> work
> > > > >>>>>>>>>>>>>>>> for
> > > > >>>>>>>>>>>>>>>>>>> you?.
> > > > >>>>>>>>>>>>>>>>>>> There are many examples of 3.5.X being used in
> > > > >>>> productions
> > > > >>>>>>>> from
> > > > >>>>>>>>>> my
> > > > >>>>>>>>>>>>>>>>> limited
> > > > >>>>>>>>>>>>>>>>>>> knowledge.
> > > > >>>>>>>>>>>>>>>>>>> ThanksPowell.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> On Wednesday, March 16, 2016 2:44 AM, Flavio
> > > > >>> Junqueira <
> > > > >>>>>>>>>>>>>>>>> fpj@apache.org>
> > > > >>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> None of us expected the reconfig changes to take
> > > > >> this
> > > > >>>> long
> > > > >>>>>> to
> > > > >>>>>>>>>>>>>>>> stabilize.
> > > > >>>>>>>>>>>>>>>>>>> Until we get there, I don't think we can do
> > anything
> > > > >>>> else
> > > > >>>>>> with
> > > > >>>>>>>>>> 3.5.
> > > > >>>>>>>>>>>>>>>> The
> > > > >>>>>>>>>>>>>>>>>>> best bet we have is to work harder to bring 3.5
> > > > >> into a
> > > > >>>>>> stable
> > > > >>>>>>>>>> rather
> > > > >>>>>>>>>>>>>>>>> than
> > > > >>>>>>>>>>>>>>>>>>> trying to work around it.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> There are lots of people interested in seeing 3.5
> > > > >>>> stable,
> > > > >>>>>> and
> > > > >>>>>>>> if
> > > > >>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>> get
> > > > >>>>>>>>>>>>>>>>>>> everyone to contribute more patches and code
> > > > >> reviews,
> > > > >>> we
> > > > >>>>>>>> should
> > > > >>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>> able
> > > > >>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>> do it sooner. After all, it is a community based
> > > > >>>> effort, so
> > > > >>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> community
> > > > >>>>>>>>>>>>>>>>>>> shouldn't rely on just 2-3 people doing the work.
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>> -Flavio
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> On 15 Mar 2016, at 17:28, Chris Nauroth <
> > > > >>>>>>>>>> cnauroth@hortonworks.com>
> > > > >>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Doug, I forgot to respond to your question about
> > > > >> 3.4.
> > > > >>>>>> Since
> > > > >>>>>>>>>> 3.4 is
> > > > >>>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>> stable maintenance line, we are very
> conservative
> > > > >>> about
> > > > >>>>>>>>>>>>>>> back-porting
> > > > >>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>> it.  Our policy is to limit back-ports to
> critical
> > > > >>> bug
> > > > >>>>>> fixes
> > > > >>>>>>>> and
> > > > >>>>>>>>>>>>>>> not
> > > > >>>>>>>>>>>>>>>>>>>> introduce any new features in the 3.4 line.
> This
> > > > >> is
> > > > >>> a
> > > > >>>>>>>> matter of
> > > > >>>>>>>>>>>>>>>>> managing
> > > > >>>>>>>>>>>>>>>>>>>> risk.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> Jason, your question about release cadence is a
> > > > >> fair
> > > > >>>>>> one.  If
> > > > >>>>>>>>>> it's
> > > > >>>>>>>>>>>>>>>> any
> > > > >>>>>>>>>>>>>>>>>>>> consolation, we are now taking the approach of
> > > > >> trying
> > > > >>>> to
> > > > >>>>>>>> limit
> > > > >>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>> scope
> > > > >>>>>>>>>>>>>>>>>>>> of anything new introduced in 3.5 too.  That
> would
> > > > >>>> allow
> > > > >>>>>> us
> > > > >>>>>>>> to
> > > > >>>>>>>>>>>>>>> focus
> > > > >>>>>>>>>>>>>>>> on
> > > > >>>>>>>>>>>>>>>>>>>> stabilization: resolving blocker bugs and
> freezing
> > > > >>>> public
> > > > >>>>>>>>>> APIs.  I
> > > > >>>>>>>>>>>>>>>>> think
> > > > >>>>>>>>>>>>>>>>>>>> this will help us accelerate the releases into
> > beta
> > > > >>> and
> > > > >>>>>>>> eventual
> > > > >>>>>>>>>>>>>>> GA.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> I am new to ZooKeeper release management, so I'd
> > > > >> like
> > > > >>>> to
> > > > >>>>>> hear
> > > > >>>>>>>>>>>>>>>> thoughts
> > > > >>>>>>>>>>>>>>>>>>>> from more experienced committers and PMC members
> > > > >>> about
> > > > >>>>>> your
> > > > >>>>>>>>>>>>>>> proposal
> > > > >>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>> try to cut a stable release for a limited subset
> > of
> > > > >>>> what
> > > > >>>>>> is
> > > > >>>>>>>> in
> > > > >>>>>>>>>>>>>>>>> branch-3.5
> > > > >>>>>>>>>>>>>>>>>>>> now.  My instinct is that it would be
> challenging
> > > > >> to
> > > > >>>>>>>> cherry-pick
> > > > >>>>>>>>>>>>>>> out
> > > > >>>>>>>>>>>>>>>>>>>> pieces of branch-3.5 piecemeal at this point.
> > This
> > > > >>>> would
> > > > >>>>>>>> become
> > > > >>>>>>>>>>>>>>>>> another
> > > > >>>>>>>>>>>>>>>>>>>> release line for an already resource-constrained
> > > > >>>> volunteer
> > > > >>>>>>>>>> staff to
> > > > >>>>>>>>>>>>>>>>>>>> manage.  I'd prefer to dedicate those limited
> > > > >>>> resources to
> > > > >>>>>>>>>> overall
> > > > >>>>>>>>>>>>>>>> 3.5
> > > > >>>>>>>>>>>>>>>>>>>> stabilization.  Also, a 3.5 release in which
> > > > >> certain
> > > > >>>>>> features
> > > > >>>>>>>>>>>>>>>>> "vanished"
> > > > >>>>>>>>>>>>>>>>>>>> because of not meeting some stability criteria
> > > > >> would
> > > > >>> be
> > > > >>>>>>>>>>>>>>> undesirable.
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> --Chris Nauroth
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>> On 3/15/16, 10:12 AM, "Jason Rosenberg" <
> > > > >>>> jbr@squareup.com
> > > > >>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Chris,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Can you say whether some parts of 3.5.X are
> more
> > > > >>>> stable
> > > > >>>>>> than
> > > > >>>>>>>>>>>>>>> others
> > > > >>>>>>>>>>>>>>>>>>> (e.g.
> > > > >>>>>>>>>>>>>>>>>>>>> if we don't care about certain new features, is
> > it
> > > > >>>>>>>> relatively
> > > > >>>>>>>>>>>>>>>> stable)?
> > > > >>>>>>>>>>>>>>>>>>>>> Would it be possible to cut out a version that
> > > > >> only
> > > > >>>> has
> > > > >>>>>> the
> > > > >>>>>>>>>> bits
> > > > >>>>>>>>>>>>>>> we
> > > > >>>>>>>>>>>>>>>>>>> think
> > > > >>>>>>>>>>>>>>>>>>>>> are stable (and release that)?
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> From that timeline, and the historic release
> > > > >>> cadence,
> > > > >>>> it
> > > > >>>>>>>> would
> > > > >>>>>>>>>>>>>>> seem
> > > > >>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>> be
> > > > >>>>>>>>>>>>>>>>>>>>> a
> > > > >>>>>>>>>>>>>>>>>>>>> years away before we get to the stable release?
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Thanks,
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> Jason
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>> On Tue, Mar 15, 2016 at 1:06 PM, Chris Nauroth
> <
> > > > >>>>>>>>>>>>>>>>>>> cnauroth@hortonworks.com>
> > > > >>>>>>>>>>>>>>>>>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Hello Doug,
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> Thanks for your interest in the SSL feature!
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> At this point, I think we're still pretty far
> > > > >> away
> > > > >>>> from
> > > > >>>>>>>>>>>>>>> declaring a
> > > > >>>>>>>>>>>>>>>>>>>>>> stable
> > > > >>>>>>>>>>>>>>>>>>>>>> release in the 3.5 line.  I don't think we're
> > > > >> close
> > > > >>>>>> enough
> > > > >>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>> anyone
> > > > >>>>>>>>>>>>>>>>>>>>>> can
> > > > >>>>>>>>>>>>>>>>>>>>>> offer a reliable ETA.  This is an earlier
> thread
> > > > >>> that
> > > > >>>>>>>>>> describes
> > > > >>>>>>>>>>>>>>> the
> > > > >>>>>>>>>>>>>>>>>>>>>> high-level strategy for release planning in
> the
> > > > >> 3.5
> > > > >>>>>> line:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> https://s.apache.org/ADK1
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> The next step is a 3.5.2-alpha release.  We're
> > > > >>>> working
> > > > >>>>>> on
> > > > >>>>>>>>>>>>>>>> resolving a
> > > > >>>>>>>>>>>>>>>>>>>>>> few
> > > > >>>>>>>>>>>>>>>>>>>>>> more blockers before we produce a release
> > > > >>> candidate.
> > > > >>>>>>>>>> Hopefully
> > > > >>>>>>>>>>>>>>>> that
> > > > >>>>>>>>>>>>>>>>>>>>>> will
> > > > >>>>>>>>>>>>>>>>>>>>>> get done in the next few weeks.
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> --Chris Nauroth
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>> On 3/15/16, 9:39 AM, "Doug" <
> > itsbehind@gmail.com
> > > > >>>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> I know it's only been a few months, but I was
> > > > >>>>>> wondering if
> > > > >>>>>>>>>> there
> > > > >>>>>>>>>>>>>>>>> was a
> > > > >>>>>>>>>>>>>>>>>>>>>>> ballpark release date for a stable version of
> > > > >>> 3.5.1.
> > > > >>>>>> Or is
> > > > >>>>>>>>>> there
> > > > >>>>>>>>>>>>>>>> any
> > > > >>>>>>>>>>>>>>>>>>>>>>> chance
> > > > >>>>>>>>>>>>>>>>>>>>>>> the SSL feature would be added to 3.4.8? Just
> > > > >>>> another
> > > > >>>>>>>> person
> > > > >>>>>>>>>>>>>>>> looking
> > > > >>>>>>>>>>>>>>>>>>> to
> > > > >>>>>>>>>>>>>>>>>>>>>>> have
> > > > >>>>>>>>>>>>>>>>>>>>>>> that feature in a stable version. Thanks for
> > all
> > > > >>> you
> > > > >>>>>> do!
> > > > >>>>>>>> :)
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>> --
> > > > >>>>>>>>>>>>>>>>>>>>>>> View this message in context:
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > >
> >
> http://zookeeper-user.578899.n2.nabble.com/Zookeeper-with-SSL-release-dat
> > > > >>>>>>>>>>>>>>>>>>>>>> e
> > > > >>>>>>>>>>>>>>>>>>>>>>> -tp7581744p7582136.html
> > > > >>>>>>>>>>>>>>>>>>>>>>> Sent from the zookeeper-user mailing list
> > > > >> archive
> > > > >>> at
> > > > >>>>>>>>>> Nabble.com.
> > > > >>>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

--089e0115f8c6aa7031052f6f248a--