accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christopher <>
Subject Re: Config in ZooKeeperInstance
Date Wed, 31 Jul 2013 20:54:16 GMT
I'm not sure I see much point implementing server-side features to
support a "use SSL, if available" client mode. Such a feature is
effectively no more secure than the basic "insecure" mode, and as
such, I think is a waste of resources to implement (and, more
importantly, to maintain). Besides, clients can implement a "use SSL,
if available" feature (if they really must) without the server side
revealing any information about how it is configured (your suggested
"try, then fallback on failure" would work, and it wouldn't be too
bad, if you did it once per "Instance"

The "connection string" for Accumulo *is* client configuration, so
that seems the most sensible route to me.
For example, one of:

// one of these could respect the JSSE system properties
ZooKeeperInstance(instanceName, zookeepers,
secureMode).getConnector(user, token);
SecureZooKeeperInstance(instanceName, zookeepers).getConnector(user, token);
Instance(instanceName, zookeepers).getSslConnector(user, token);

On the other hand, if you mean per-tserver connection strings
(locations in metadata or server advertisements/locks in ZK), I'd
steer clear, because you really need such a setting consistent for
everything related to an entire o.a.a.core.client.Instance. If even
one tserver is insecure and operating within the cluster, and the
client is configured to fallback to insecure mode on a per-connection
basis, then the whole system is insecure.

You could implement a ZK flag that reports which mode (set by the
Master) the entire instance is running in, but I think this should be
used only as a fast-failure when constructing a secure client
connector. Otherwise, if the client is going to change its operation
mode based on this flag, you're going to have to set up an additional
layer of trust with ZK before you can trust that it's safe to respect
this flag in the first place (because the flag has the possibility of
forcing an insecure mode).

Setting up that trust seems complicated and unnecessary to me (I
really don't like the idea of an insecure fallback mode), but it might
be worth it for other reasons anyway. I'm not sure what options are
available to lock down ZK more. Perhaps an additional shared secret
for clients (again, distributed as client configuration)? or
client-certificate authentication for the read-only client areas of

I suggest:

Have the master set a flag in ZK in an area that the client can read,
to inform the client which mode the server is operating in, for
fast-failure only (to enable client logic to fall back, if desired).
Provide an Instance that will use SSL mode for all its RPC calls, and
will fail if the server's flag doesn't match the specified mode.
Make clients configurable, to use either the secure Instance or the
insecure Instance (do this entirely in client code).
Do additional research on securing/trusting ZK itself.

Christopher L Tubbs II

On Wed, Jul 31, 2013 at 11:52 AM, Michael Berman <> wrote:
> Oh, I absolutely agree we can't trust remote config to tell us where to
> find the trusted root...that needs to be consciously configured for every
> client.  I was thinking more along the lines of your "http can
> automatically redirect to https" example.  If I have an accumulo cluster up
> and running, and I want to switch to SSL, it would be nice to be able to
> flip the switch on the server, distribute my root cert, and then have all
> the existing client apps able to figure out that they should make SSL
> connections to accumulo automatically based on some cue from ZK.  If it's
> really entirely up to the client to decide, probably that switch would
> require a recompile of my client apps to add the "use SSL" flag.  I
> wouldn't want that flag to be in a conf file (although I think that's a
> great place for the path to the root cert), because on any given machine I
> may want to connect to both SSL and non-SSL seems like a
> connection-specific setting, not a global machine setting.  I do definitely
> think there's a place in the client interface for "require SSL" in,
> I'm only interested in connecting to accumulo if I can get an SSL
> connection.  But I think if that's not set, it would be nice for the
> behavior to be "I don't care" rather than "require non-SSL", in which case
> we do need some hint from ZK.  You have convinced me that we don't want the
> whole config in there, so maybe it should just be a component of the
> connection string, or a standalone flag.  I suppose another option is to
> always try SSL first, and if that fails, fall back to non-SSL, but that
> seems like a lot of overhead every time we try to make a connection,
> especially if the common case is not SSL.
> On Tue, Jul 23, 2013 at 5:02 PM, Christopher <> wrote:
>> It should really be a conscious decision whether to create a client
>> that expects SSL enabled or not. It shouldn't just happen
>> automatically.
>> For comparison, imagine if HTTP/SSL worked this way... when I type
>> https://.... I expect it to be serving over SSL... not just
>> auto-negotiate to secure or non-secure mode. If it's not in secure
>> mode when I navigate to https://... , I simply don't want to go there.
>> Now, browsers do auto-negotiate in the other direction, to secure mode
>> (via HTTP redirects to HTTPS addresses), but that only works because
>> your browser has a set of built-in certificate authorities that it
>> trusts. If it doesn't, your browser gives you an error (if it's a good
>> browser) or a warning (if it's a weak one). So, even auto-negotiating
>> to secure mode is problematic if you can't control from the client,
>> which certificates the client is willing to trust in that secure
>> communication. We have no sensible way to distribute certificates that
>> clients should trust. Sysadmins should generate their own certificates
>> and configure clients appropriately.
>> So, I don't see a way around it, you're going to have to make the
>> client code aware that SSL is the desired mode of communication...
>> either through a constructor parameter to ZooKeeperInstance, or
>> through another Instance sub-class... perhaps one that reads a
>> configuration file from the user's home directory with standard JSSE
>> properties (
>> ).
>> You should not just automatically retrieve this from the server's
>> configuration in ZooKeeper, because that assumes you already trust the
>> server, and that undermines the whole point of the authentication
>> component of SSL.
>> There's also the question of enabling SSL on ZK itself. I think you'd
>> probably want to do that, too. You're still going to need to seed the
>> SSL attributes in the client configuration, though, before you talk to
>> any other server. I suppose you could trust ZK, and manage keys there
>> (still seeding the client, but only with the certs / ssl mode needed
>> to talk directly to ZK)... but I think that might be overkill for a
>> first pass (especially since you can probably re-use the same server
>> cert for ZK as you can for the other servers... since they are
>> operating as a system). I think a simpler and more reliable solution
>> is to enable the easy configuration of standard JSSE properties in
>> both the client and the server.
>> Now, what we could do is leave the configuration getter/setter on the
>> instance, and require setting JSSE options for SSL in that, via
>> setConfiguration(), though I'm personally a fan of getting them
>> automatically from a properties file instead (like the
>> $HOME/.accumulorc or $HOME/.accumulo/config suggestion made on
>> ACCUMULO-1045, or maybe even /etc/accumulo/clients.conf or similar).
>> --
>> Christopher L Tubbs II
>> On Tue, Jul 23, 2013 at 1:38 PM, Michael Berman <> wrote:
>> > The reason I'm interested in the config from a ZooKeeperInstance is
>> because
>> > a ZKInstance needs to know whether or not to use SSL to try to connect
>> to a
>> > given cluster.  I think it would be annoying to require a ZKInstance
>> client
>> > to explicitly specify "useSsl" any time they want to connect to an SSL
>> > tserver (although I do see value in having a "requireSsl" optional flag).
>> >  If configuration isn't the right way to communicate this, do people have
>> > other suggestions?  I suppose I could just throw an ssl:// on the front
>> of
>> > the connection string or something, but it seems kludgy to do that any
>> time
>> > there's a configuration option that we want clients to know about.
>> >
>> > As far as removing Instance.getConfiguration() entirely, there are about
>> 30
>> > references throughout the codebase to getConfiguration() on a bare
>> > Instance.  Finding some other way to plumb whatever options those
>> consumers
>> > care about through the stack sounds like a substantial project.
>> >
>> > In my use case, I am only interested in the configuration that's stored
>> in
>> > ZK, and I already have enough information to connect to ZK, so it doesn't
>> > seem difficult technically, just a question of desired behavior.
>> >
>> >
>> > On Tue, Jul 23, 2013 at 1:11 PM, Eric Newton <>
>> wrote:
>> >
>> >> +1 for removing it from Instance
>> >>
>> >>
>> >> On Tue, Jul 23, 2013 at 12:14 PM, Christopher <>
>> wrote:
>> >>
>> >> > It only returns the default config if you don't call
>> >> > setConfiguration(), which appears to be almost always (except
>> >> > TestIngest).
>> >> >
>> >> > I don't know that this API is clearly spelled out, as to its intended
>> >> > purpose. Which configuration is it supposed to be getting, and how
>> >> > does that relate to the ZooKeeperInstance? The only configuration a
>> >> > ZooKeeperInstance has is the minimum needed to connect to other
>> >> > servers. It still has to authenticate to read any other server
>> >> > configuration.
>> >> >
>> >> > Personally, I'd be in favor of removing it from Instance interface,
>> >> > unless we actually document what it is supposed to be for to justify
>> >> > its utility in, and relationship to, the Instance interface.
>> >> >
>> >> > --
>> >> > Christopher L Tubbs II
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Jul 23, 2013 at 10:44 AM, Michael Berman <>
>> >> > wrote:
>> >> > > I was surprised to find that ZooKeeperInstance.getConfiguration()
>> seems
>> >> > to
>> >> > > return only the default configuration rather than the configuration
>> >> > stored
>> >> > > in ZK.  Is this the expected behavior?  I came across this in
>> >> MacTest,
>> >> > so
>> >> > > if other people are also surprised, it may be a MAC or
>> MacTest-specific
>> >> > > issue, in which case I'm happy to track it down.
>> >> > >
>> >> > > If this is currently the expected behavior, how would people feel
>> about
>> >> > > changing it?  It seems like it would be useful to have a config
>> channel
>> >> > to
>> >> > > ZooKeeperInstance clients (in my case, what I'm specifically
>> interested
>> >> > in
>> >> > > is whether or not SSL is enabled).  There may be a potential for
>> >> > privileged
>> >> > > information to escape...table settings, for example, may be
>> >> sensitive...
>> >> > > But all the really secret stuff should be in the site.xml which
>> >> wouldn't
>> >> > > get exposed.
>> >> > >
>> >> > > For reference, ZooKeeperInstance's getConfiguration() is implemented
>> >> as:
>> >> > > AccumuloConfiguration.getDefaultConfiguration()
>> >> > >
>> >> > > whereas HdfsZooInstance's getConfiguration() is:
>> >> > > ZooConfiguration.getInstance(this, getSiteConfiguration())
>> >> > >
>> >> > > My proposal would be to change ZooKeeperInstance's to something
>> like:
>> >> > > ZooConfiguration.getInstance(this,
>> >> > > AccumuloConfiguration.getDefaultConfiguration())
>> >> > >
>> >> > > Michael
>> >> >
>> >>

View raw message