kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin McCabe" <cmcc...@apache.org>
Subject Re: [DISCUSS] KIP-515: Enable ZK client to use the new TLS supported authentication
Date Wed, 15 Jan 2020 20:52:59 GMT
On Wed, Jan 15, 2020, at 10:53, Ron Dagostino wrote:
> Thanks Colin and Rajini.
> 
> I've updated the KIP to say that KIP-421 functionality is available to
> encrypt sensitive configs like the ZK key store and trust store
> passwords.  (I've also made it clear that the configs are not
> dynamically reconfigurable since dynamic values are stored in ZK and
> the data is needed to get to ZK in the first place).  Colin, let me
> know if you think anything else needs to be said/done about it -- I
> wasn't sure if your comment above implies that there are additional
> direct actions that we need to take aside from this.
> 
> I agree that making the brokers' ZooKeeper clients dynamic with
> respect to key and trust stores (e.g. via
> zookeeper.ssl.context.supplier.class) may increase the risk that this
> KIP may not land in time for 2.5.
> 
> Regarding the config names for ZK being different than the ZooKeeper
> configs (e.g. camel-case keyStore/trustStore instead of
> keystore/truststore, ciphersuites instead of cipher.suites,
> enabledProtocols instead of enabled.protocols), I agree it is an
> issue.  I tried to mitigate it by putting warning text in the config
> docs for these.  Regarding configuration inheritance, I think what you
> are saying is that for any configs where the broker has a clear
> semantic equivalent, the broker could fall back to the broker value if
> the ZK value is not given.  The potential list is:
> 
> zookeeper.ssl.keyStore.location => ssl.keystore.location
> zookeeper.ssl.keyStore.password => ssl.keystore.password
> zookeeper.ssl.keyStore.type => ssl.keystore.type
> zookeeper.ssl.trustStore.location => ssl.truststore.location
> zookeeper.ssl.trustStore.password => ssl.truststore.password
> zookeeper.ssl.trustStore.type => ssl.truststore.type
> zookeeper.ssl.ciphersuites => ssl.cipher.suites
> 
> Note that there are two configs that look the same based on their key
> names but whose allowable values differ:
> 
> zookeeper.ssl.protocol(Default: TLSv1.2) => ssl.protocol(Default: TLS)
> zookeeper.ssl.enabledProtocols(Default: value of protocol property) =
> ssl.enabled.protocols(Default: TLSv1.2,TLSv1.1,TLSv1)
> 
> Not sure what the right answer is, but hopefully this detail will help
> get us there.
> 
> Ron

I think, on balance, I agree with Rajini that it would be nice to make the configs look more
like Kafka configs.  We don't have camel-case in configuration keys, so we should avoid it
here.

best,
Colin

> 
> On Wed, Jan 15, 2020 at 12:26 PM Colin McCabe <cmccabe@apache.org> wrote:
> >
> > On Tue, Jan 14, 2020, at 11:49, Rajini Sivaram wrote:
> > > Hi Ron,
> > >
> > > Thanks for the detailed explanation, sounds good to me.
> > >
> > > A few more questions:
> > >
> > > 1) At the moment, all sensitive broker configs including
> > > keystore/truststore passwords can be stored encrypted in ZooKeeper prior to
> > > starting up brokers. We are now adding SSL keystore/truststore passwords
> > > for ZooKeeper client that cannot be stored in ZooKeeper since you need
> > > these to connect to ZK. We should perhaps document that these configs can
> > > be encrypted using KIP-421.
> >
> > That's a good point.  Previously, we didn't have ZK on-the-wire security support.
 However, now that we do, sending sensitive keystore and truststore passwords to ZK in cleartext
should be deprecated in favor of using KIP-421.
> >
> > >
> > > 2) We can dynamically update keystores and trust stores used by brokers
> > > without restarting the broker. Can we support this easily for ZK clients
> > > created by the broker, for example by adding our own
> > > `zookeeper.ssl.context.supplier.class`?
> > >
> >
> > Hmm.  That might be better to consider in a follow-up, since the deadline for 2.5
KIPs is coming up?
> >
> > best,
> > Colin
> >
> > > 3) Looks like we are using config names that directly map to ZK configs.
> > > Have we considered using equivalent Kafka config names with prefixes,
> > > perhaps with inheritance from the non-prefixed names? Not sure if this is a
> > > good idea, but perhaps worth documenting in Rejected Alternatives at least.
> > >
> > >
> > > On Tue, Jan 14, 2020 at 5:14 PM Colin McCabe <cmccabe@apache.org> wrote:
> > >
> > > > Hi Ron,
> > > >
> > > > Thanks for the explanation.  I guess thinking about it a little bit more,
> > > > we should just add --zk-tls-config-file to all of these commands.
> > > >
> > > > We will be removing this option (plus ZK in general) from these commands
> > > > in the next major release, but ZK is still supported in 2.5, so we should
> > > > just do the logical thing.  (The exception is ZkSecurityMigrator, which
> > > > will stay).
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Jan 14, 2020, at 07:38, Ron Dagostino wrote:
> > > > > Hi Colin.
> > > > >
> > > > > <<< It seems like this [--zk-tls-config-file information]
could just
> > > > appear
> > > > > in a configuration file, which all of these tools already accept
(I
> > > > think)
> > > > >
> > > > > ZkSecurityMigrator has no such property file facility; adding a
> > > > > "--zk-tls-config-file" parameter is exactly for this purpose.  If
we add
> > > > > that to ZkSecurityMigrator then it is trivial to add it to other
commands
> > > > > (the same code is simply reused; it ends up being just a few extra
> > > > lines).
> > > > > I do not see any parameter in the other two commands to adjust the
ZK
> > > > > connection; ConfigCommand accepts a "--command-config" flag, but
> > > > according
> > > > > to the code "This is used only with --bootstrap-server option for
> > > > > describing and altering broker configs."
> > > > >
> > > > > I do agree there would be no need to add "--zk-tls-config-file" to
> > > > > ReassignPartitionsCommand if its direct ZK connectivity is replaced
in
> > > > time
> > > > > for the next release.
> > > > >
> > > > > ConfigCommand supports the "--bootstrap-server" option and will have
its
> > > > > direct ZooKeeper access formally deprecated as per KIP-555, but the
> > > > special
> > > > > use case of bootstrapping a ZooKeeper ensemble with encrypted credentials
> > > > > prior to starting Kafka will still exist, so it feels like while
> > > > > "--zk-tls-config-file" would never be used except for this use case
it
> > > > > could probably still be added for this particular situation.
> > > > >
> > > > > Ron
> > > > >
> > > > > P.S. I responded on 1/6 but I just discovered that e, ail (and 3
more I
> > > > > sent) did not go through.  I am trying to get emails through now
to move
> > > > > this discussion forward.
> > > > >
> > > > > On Mon, Jan 6, 2020 at 5:07 PM Colin McCabe <cmccabe@apache.org>
wrote:
> > > > >
> > > > > > On Fri, Dec 27, 2019, at 10:48, Ron Dagostino wrote:
> > > > > > > Hi everyone.  I would like to make the following changes
to the KIP.
> > > > > > >
> > > > > > > MOTIVATION:
> > > > > > > Include a statement that it will be difficult in the short
term to
> > > > > > > deprecate direct Zookeeper communication in kafka-configs.{sh,
bat}
> > > > > > (which
> > > > > > > invoke kafka.admin.ConfigCommand) because bootstrapping
a Kafka
> > > > cluster
> > > > > > > with encrypted passwords in Zookeeper is an explicitly-supported
use
> > > > > > case;
> > > > > > > therefore it is in scope to be able to securely configure
the CLI
> > > > tools
> > > > > > > that still leverage non-deprecated direct Zookeeper communication
> > > > for TLS
> > > > > > > (the other 2 tools are kafka-reassign-partitions.{sh, bat}
and
> > > > > > > zookeeper-security-migration.sh).
> > > > > >
> > > > > > Hi Ron,
> > > > > >
> > > > > > Thanks for the KIP.
> > > > > >
> > > > > > About deprecations:
> > > > > >
> > > > > > * zookeeper-security-migration: clearly, deprecating ZK access
in this
> > > > one
> > > > > > would not make sense, since it would defeat the whole point
of the
> > > > tool :)
> > > > > >
> > > > > > * kafka-reassign-partitions: ZK access should be deprecated
here.
> > > > KIP-455
> > > > > > implementation has dragged a bit, but this should get done soon.
> > > > Certainly
> > > > > > before the next release.
> > > > > >
> > > > > > * kafka-configs: I think ZK access should be deprecated here
as well.
> > > > I
> > > > > > agree there is a super-special bootstrapping case here, but
that should
> > > > > > have its own tool, not use kafka-configs.
> > > > > >
> > > > > > I will post a separate KIP for this, though.
> > > > > >
> > > > > > >
> > > > > > > GOALS:
> > > > > > > Support the secure configuration of TLS-encrypted communication
> > > > between
> > > > > > > Zookeeper and:
> > > > > > >   a) Kafka brokers
> > > > > > >   b) The three CLI tools mentioned above that still support
direct,
> > > > > > > non-deprecated communication to Zookeeper
> > > > > > > It is explicitly out-of-scope to deprecate any direct Zookeeper
> > > > > > > communication in CLI tools as part of this KIP; such work
will occur
> > > > in
> > > > > > > future KIPs instead.
> > > > > > >
> > > > > > > PUBLIC INTERFACES:
> > > > > > > 1) The following new broker configurations will be recognized.
> > > > > > >   zookeeper.client.secure (default value = false, for backwards
> > > > > > > compatibility)
> > > > > > >   zookeeper.clientCnxnSocket
> > > > > > >   zookeeper.ssl.keyStore.location
> > > > > > >   zookeeper.ssl.keyStore.password
> > > > > > >   zookeeper.ssl.trustStore.location
> > > > > > >   zookeeper.ssl.trustStore.password
> > > > > > > It will be an error for any of the last 5 values to be
left
> > > > unspecified
> > > > > > if
> > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > > >
> > > > > > > 2) In addition, the kafka.security.authorizer.AclAuthorizer
class
> > > > > > supports
> > > > > > > the ability to connect to a different Zookeeper instance
than the
> > > > one the
> > > > > > > brokers use.  We therefore also add the following optional
configs,
> > > > which
> > > > > > > override the corresponding ones from above when present:
> > > > > > >   authorizer.zookeeper.client.secure
> > > > > > >   authorizer.zookeeper.clientCnxnSocket
> > > > > > >   authorizer.zookeeper.ssl.keyStore.location
> > > > > > >   authorizer.zookeeper.ssl.keyStore.password
> > > > > > >   authorizer.zookeeper.ssl.trustStore.location
> > > > > > >   authorizer.zookeeper.ssl.trustStore.password
> > > > > > >
> > > > > > > 3) The three CLI tools mentioned above will support a new
> > > > > > --zk-tls-config-file
> > > > > > > <String: Zookeeper TLS configuration file path>"
option.  The
> > > > following
> > > > > > > properties will be recognized in that file, and unrecognized
> > > > properties
> > > > > > > will be ignored to allow the possibility of pointing
> > > > zk-tls-config-file
> > > > > > at
> > > > > > > the broker's config file.
> > > > > > >   zookeeper.client.secure (default value = false)
> > > > > > >   zookeeper.clientCnxnSocket
> > > > > > >   zookeeper.ssl.keyStore.location
> > > > > > >   zookeeper.ssl.keyStore.password
> > > > > > >   zookeeper.ssl.trustStore.location
> > > > > > >   zookeeper.ssl.trustStore.password
> > > > > > > It will be an error for any of the last 5 values to be
left
> > > > unspecified
> > > > > > if
> > > > > > > zookeeper.client.secure is explicitly set to true.
> > > > > >
> > > > > > Do we really need a --zk-tls-config-file flag?  It seems like
this
> > > > could
> > > > > > just appear in a configuration file, which all of these tools
already
> > > > > > accept (I think).
> > > > > >
> > > > > > best,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Ron
> > > > > > >
> > > > > > > On Mon, Dec 23, 2019 at 3:03 PM Ron Dagostino <rndgstn@gmail.com>
> > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone.  Let's get this discussion going again
now that Kafka
> > > > 2.4
> > > > > > > > with Zookeeper 3.5.6 is out.
> > > > > > > >
> > > > > > > > First, regarding the KIP number, the other KIP that
was using this
> > > > > > number
> > > > > > > > moved to KIP 534, so KIP 515 remains the correct number
for this
> > > > > > > > discussion.  I've updated the Kafka Improvement Proposal
page to
> > > > list
> > > > > > this
> > > > > > > > KIP in the 515 slot, so we're all set there.
> > > > > > > >
> > > > > > > > Regarding the actual issues under discussion, I think
there are
> > > > some
> > > > > > > > things we should clarify.
> > > > > > > >
> > > > > > > > 1) It is possible to use TLS connectivity to Zookeeper
from Apache
> > > > > > Kafka
> > > > > > > > 2.4 -- the problem is that configuration information
has to be
> > > > passed
> > > > > > via
> > > > > > > > system properties as "-D" command line options on
the java
> > > > invocation
> > > > > > of
> > > > > > > > the broker, and those are not secure (anyone with
access to the box
> > > > > > can see
> > > > > > > > the command line used to invoke the broker); the configuration
> > > > includes
> > > > > > > > sensitive information (e.g. a keystore password),
so we need a
> > > > secure
> > > > > > > > mechanism for passing the configuration values.  I
believe the real
> > > > > > > > motivation for this KIP is to harden the configuration
mechanism
> > > > for
> > > > > > > > Zookeeper TLS connectivity.
> > > > > > > >
> > > > > > > > 2) I believe the list of CLI tools that continue to
use direct
> > > > > > Zookeeper
> > > > > > > > connectivity in a non-deprecated fashion is:
> > > > > > > >   a)
> > > > zookeeper-security-migration.sh/kafka.admin.ZkSecurityMigrator
> > > > > > > >   b)
> > > > > > > >
> > > > > >
> > > > kafka-reassign-partitions.{sh,bat}/kafka.admin.ReassignPartitionsCommand
> > > > > > > >   c) kafka-configs.{sh, bat}/kafka.admin.ConfigCommand
> > > > > > > >
> > > > > > > > 3) I believe Kafka.admin.ConfigCommand presents a
conundrum
> > > > because it
> > > > > > > > explicitly states in a comment that a supported use
case is
> > > > > > bootstrapping a
> > > > > > > > Kafka cluster with encrypted passwords in Zookeeper
(see
> > > > > > > >
> > > > > >
> > > > https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ConfigCommand.scala#L65
> > > > > > ).
> > > > > > > > This means it will be especially difficult to deprecate
this
> > > > particular
> > > > > > > > direct Zookeeper connectivity without a different
storage
> > > > mechanism for
> > > > > > > > dynamic configuration values being available (i.e.
the self-managed
> > > > > > quorum
> > > > > > > > referred to in KIP-500).
> > > > > > > >
> > > > > > > > I think it would be easier and simpler to harden the
Zookeeper TLS
> > > > > > > > configuration in both Kafka and in CLI tools compared
to trying to
> > > > > > > > deprecate the direct Zookeeper connectivity in the
above 3 CLI
> > > > tools --
> > > > > > > > especially when ConfigCommand has no obvious short-term
path to
> > > > > > deprecation.
> > > > > > > >
> > > > > > > > Regarding how to harden the TLS configuration, adding
the
> > > > appropriate
> > > > > > > > configs to Kafka is an obvious choice for the brokers.
 Not that
> > > > these
> > > > > > > > values cannot be dynamically reconfigurable because
the values
> > > > would be
> > > > > > > > required to connect to Zookeeper via TLS in the first
place.
> > > > > > > >
> > > > > > > > Regarding how to harden the TLS configuration for
the 3 remaining
> > > > CLI
> > > > > > > > tools, it would be relatively straightforward to add
support for a
> > > > > > > > "--zk-tls-config-file <String: Zookeeper TLS configuration
system
> > > > > > > > properties file path>" option to each one.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > On Thu, Sep 12, 2019 at 8:13 AM Pere Urbón Bayes
<
> > > > pere.urbon@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> True, the number is duplicated.
> > > > > > > >>
> > > > > > > >>   do you know how can we get the problem fix?
I was not aware, I
> > > > > > missed,
> > > > > > > >> sorry the need to add the KIP to the table.
> > > > > > > >>
> > > > > > > >> -- Pere
> > > > > > > >>
> > > > > > > >> Missatge de Jorge Esteban Quilcate Otoya <
> > > > quilcate.jorge@gmail.com>
> > > > > > del
> > > > > > > >> dia
> > > > > > > >> dt., 3 de set. 2019 a les 13:43:
> > > > > > > >>
> > > > > > > >> > Hi Pere,
> > > > > > > >> >
> > > > > > > >> > Have you add your KIP to the list here
> > > > > > > >> >
> > > > > > > >>
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> > > > > > > >> ?
> > > > > > > >> >
> > > > > > > >> > I found the KIP number assigned to another.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Mon, Sep 2, 2019 at 2:23 PM Pere Urbón
Bayes <
> > > > > > pere.urbon@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >> Thanks for your time Harsha,
> > > > > > > >> >>    anyone else with comments? looking
forward to hearing from
> > > > you.
> > > > > > > >> >>
> > > > > > > >> >> Stupid question: when do you move from
discussion to vote?
> > > > > > > >> >>
> > > > > > > >> >> Missatge de Harsha Chintalapani <kafka@harsha.io>
del dia
> > > > dv., 30
> > > > > > > >> d’ag.
> > > > > > > >> >> 2019 a les 21:59:
> > > > > > > >> >>
> > > > > > > >> >> > Thanks Pere. KIP looks good to me.
> > > > > > > >> >> > -Harsha
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >> > On Fri, Aug 30, 2019 at 10:05 AM,
Pere Urbón Bayes <
> > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > >> >> > wrote:
> > > > > > > >> >> >
> > > > > > > >> >> >> Not really,
> > > > > > > >> >> >>   my idea is to keep the JAAS
parameter, so people don't see
> > > > > > major
> > > > > > > >> >> >> changes. But if you pass a properties
file, then this takes
> > > > > > > >> precedence
> > > > > > > >> >> over
> > > > > > > >> >> >> the other, with the idea that
you can do sasl as well with
> > > > the
> > > > > > > >> >> properties
> > > > > > > >> >> >> files.
> > > > > > > >> >> >>
> > > > > > > >> >> >> Makes sense?
> > > > > > > >> >> >>
> > > > > > > >> >> >> -- Pere
> > > > > > > >> >> >>
> > > > > > > >> >> >> Missatge de Harsha Chintalapani
<kafka@harsha.io> del dia
> > > > dv.,
> > > > > > 30
> > > > > > > >> >> d’ag.
> > > > > > > >> >> >> 2019 a les 19:00:
> > > > > > > >> >> >>
> > > > > > > >> >> >>> Hi Pere,
> > > > > > > >> >> >>>           Thanks for the
KIP. Enabling SSL for zookeeper
> > > > for
> > > > > > Kafka
> > > > > > > >> >> makes
> > > > > > > >> >> >>> sense.
> > > > > > > >> >> >>> "The changes are planned
to be introduced in a compatible
> > > > way,
> > > > > > by
> > > > > > > >> >> >>> keeping the current JAAS
variable precedence."
> > > > > > > >> >> >>> Can you elaborate a bit
here. If the user configures a JAAS
> > > > > > file
> > > > > > > >> with
> > > > > > > >> >> >>> Client section it will take
precedence over zookeeper SSL
> > > > > > configs?
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> Thanks,
> > > > > > > >> >> >>> Harsha
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>> On Fri, Aug 30, 2019 at
7:50 AM, Pere Urbón Bayes <
> > > > > > > >> >> pere.urbon@gmail.com>
> > > > > > > >> >> >>> wrote:
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>> Hi,
> > > > > > > >> >> >>>> quick question, I saw
in another mail that 2.4 release is
> > > > > > planned
> > > > > > > >> for
> > > > > > > >> >> >>>> September. I think it
would be really awesome to have
> > > > this for
> > > > > > > >> this
> > > > > > > >> >> >>>> release, do you think
we can make it?
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> -- Pere
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Missatge de Pere Urbón
Bayes <pere.urbon@gmail.com> del
> > > > dia
> > > > > > dj.,
> > > > > > > >> 29
> > > > > > > >> >> >>>> d’ag. 2019 a les 20:10:
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Hi,
> > > > > > > >> >> >>>> this is my first KIP
for a change in Apache Kafka, so I'm
> > > > > > really
> > > > > > > >> need
> > > > > > > >> >> >>>> to the process. Looking
forward to hearing from you and
> > > > learn
> > > > > > the
> > > > > > > >> >> best
> > > > > > > >> >> >>>> ropes here.
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> I would like to propose
this KIP-515 to enable the
> > > > > > > >> ZookeeperClients
> > > > > > > >> >> to
> > > > > > > >> >> >>>> take full advantage
of the TLS communication in the new
> > > > > > Zookeeper
> > > > > > > >> >> 3.5.5.
> > > > > > > >> >> >>>> Specially interesting
it the Zookeeper Security Migration,
> > > > > > that
> > > > > > > >> >> without
> > > > > > > >> >> >>>> this change will not
work with TLS, disabling users to use
> > > > > > ACLs
> > > > > > > >> when
> > > > > > > >> >> the
> > > > > > > >> >> >>>> Zookeeper cluster use
TLS.
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> link:
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>>
> > > > > > > >> >>
> > > > > > > >>
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> Looking forward to hearing
from you on this,
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> /cheers
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> --
> > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > >> >> >>>> Software Architect
> > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>> --
> > > > > > > >> >> >>>> Pere Urbon-Bayes
> > > > > > > >> >> >>>> Software Architect
> > > > > > > >> >> >>>> http://www.purbon.com
> > > > > > > >> >> >>>> https://twitter.com/purbon
> > > > > > > >> >> >>>> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>>
> > > > > > > >> >> >>
> > > > > > > >> >> >> --
> > > > > > > >> >> >> Pere Urbon-Bayes
> > > > > > > >> >> >> Software Architect
> > > > > > > >> >> >> http://www.purbon.com
> > > > > > > >> >> >> https://twitter.com/purbon
> > > > > > > >> >> >> https://www.linkedin.com/in/purbon/
> > > > > > > >> >> >>
> > > > > > > >> >> >
> > > > > > > >> >> >
> > > > > > > >> >>
> > > > > > > >> >> --
> > > > > > > >> >> Pere Urbon-Bayes
> > > > > > > >> >> Software Architect
> > > > > > > >> >> http://www.purbon.com
> > > > > > > >> >> https://twitter.com/purbon
> > > > > > > >> >> https://www.linkedin.com/in/purbon/
> > > > > > > >> >>
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Pere Urbon-Bayes
> > > > > > > >> Software Architect
> > > > > > > >> http://www.purbon.com
> > > > > > > >> https://twitter.com/purbon
> > > > > > > >> https://www.linkedin.com/in/purbon/
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

Mime
View raw message