Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
From: Jordan Zimmerman <jordan@jordanzimmerman.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_DEF495AE-E72E-4A77-9520-A6E5B954F9B9"
Mime-Version: 1.0 (Mac OS X Mail 10.2 \(3259\))
Subject: Re: etcd performance comparison
Date: Wed, 22 Feb 2017 16:49:54 -0300
References: <CAFp_Nis+pzKna-g65YecgVWLFXQsLN37kHMykWSjTNqQpgq1aA@mail.gmail.com>
 <CACobuG1Qn_Gw1TUbzjsiAhcznEB6XPvjWSGtKEtfEoJ30oEF9A@mail.gmail.com>
 <CA+Vv-uj9ZOussE_cA9qzYA5ZRTrsgKT8dw5y-4doD62wPGna4A@mail.gmail.com>
 <82099c83-186e-e5ba-f18d-3b4f78785386@wingcon.com>
 <CABWqe2boArd1Ra828YUGwQ1s44JgvV9K9gf4xc0cVrU+afkJYA@mail.gmail.com>
 <B346A2AF-8B86-4F36-98D8-3A527833505C@jordanzimmerman.com>
 <CABWqe2ZwLx1e=8t1vR0Ox4ycCbOQ-XqRyUUSaWM+1V7QjxFi5w@mail.gmail.com>
 <CACobuG3MGUbX_+6NL_b7t==GP1UzgDL_YEmM02XFh+_s_Tx-fw@mail.gmail.com>
To: user@zookeeper.apache.org
In-Reply-To: <CACobuG3MGUbX_+6NL_b7t==GP1UzgDL_YEmM02XFh+_s_Tx-fw@mail.gmail.com>
Message-Id: <04DF32D6-F287-4A32-A397-1494554C2287@jordanzimmerman.com>
archived-at: Wed, 22 Feb 2017 19:50:04 -0000

--Apple-Mail=_DEF495AE-E72E-4A77-9520-A6E5B954F9B9
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

https://issues.apache.org/jira/browse/ZOOKEEPER-2703 =
<https://issues.apache.org/jira/browse/ZOOKEEPER-2703>


> On Feb 22, 2017, at 2:40 PM, Dan Benediktson =
<dbenediktson@twitter.com.INVALID> wrote:
>=20
> Performance benchmarking is a very hard problem, so let's keep that in =
mind
> before criticizing this one overmuch, and before going too far trying =
to
> build our own. I do agree that the benchmark chosen here is probably =
not
> the most useful in guiding customers to select among their options for
> coordination databases, so I like Jordan's suggestion: first define a =
small
> number of interesting benchmarks, based on common use cases for these
> coordination databases. On the topic of service discovery, I agree =
that's
> probably the #1 use case, so a benchmark trying to replicate that =
scenario
> would likely be the first and most important one to go after.
>=20
> To be honest, I would expect all existing ZK releases to perform much
> worse, by comparison, to etcd and consul, with any kind of mixed read =
and
> write workload, and I think it would would help demonstrate the =
benefits of
> the patch that recently landed in trunk, and any other subsequent
> performance-oriented patches we might go after, if we had some ready
> benchmarks which could clearly demonstrate the beneficial results of =
those
> patches.
>=20
> Thanks,
> Dan
>=20
> On Wed, Feb 22, 2017 at 6:52 AM, Camille Fournier <camille@apache.org>
> wrote:
>=20
>> Even just writing about what objective tests might look like would be =
a
>> good start! I'm happy to read draft posts by anyone who wishes to =
write on
>> the topic.
>>=20
>> C
>>=20
>> On Wed, Feb 22, 2017 at 9:36 AM, Jordan Zimmerman <
>> jordan@jordanzimmerman.com> wrote:
>>=20
>>> IMO there is tremendous FUD in the etcd world. It's the new cool toy =
and
>>> ZK feels old. To suggest that ZK does not do Service Discovery is
>>> ludicrous. That was one of the very first Curator recipes.
>>>=20
>>> It might be useful to counter this trend objectively. I'd be =
interested
>> in
>>> helping. Anyone else? We can create objective tests that compare =
common
>> use
>>> cases.
>>>=20
>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>>> Jordan Zimmerman
>>>=20
>>>> On Feb 22, 2017, at 11:21 AM, Camille Fournier <camille@apache.org>
>>> wrote:
>>>>=20
>>>> I think that my biggest feeling about this blog post (besides not
>>>> disclosing the disk setup clearly) is that, ZK is really not =
designed
>> to
>>>> have massive write throughput. I would not traditionally recommend
>>> someone
>>>> use ZK in that manner. If we think that evolving it to be useful =
for
>> such
>>>> workloads would be good, it could be an interesting community
>> discussion,
>>>> but it's really not the purpose of the system design.
>>>>=20
>>>> I'd love to see a more read/write mixed load test for the systems, =
as
>>> well
>>>> as a blog post about why you might choose different systems for
>> different
>>>> workloads. I think developers have a hard time really understanding =
the
>>>> tradeoffs they are choosing in these systems, because of the nuance
>>> around
>>>> them.
>>>>=20
>>>> For me, I'm more concerned about the fact that I saw a talk =
yesterday
>>> that
>>>> mentioned both etcd and consul as options for service discovery but =
not
>>> ZK.
>>>> That feels like a big hit for our community. Orthogonal to this =
topic,
>>> just
>>>> feels worth mentioning.
>>>>=20
>>>> C
>>>>=20
>>>> On Wed, Feb 22, 2017 at 4:05 AM, Alexander Binzberger <
>>>> alexander.binzberger@wingcon.com> wrote:
>>>>=20
>>>>> 1. Seams like it might make sense to increase snapCount for those
>> tests.
>>>>>=20
>>>>> 2. ZK write performance also depends on the number of watches - =
afaik.
>>>>> This is not mentioned and not tested.
>>>>>=20
>>>>> 3. Does it really make sense to "blast" the store? Wouldn't it =
make
>> more
>>>>> sense to compare fixed write/read per clients rates?
>>>>>=20
>>>>>=20
>>>>>=20
>>>>>> Am 22.02.2017 um 05:53 schrieb Michael Han:
>>>>>>=20
>>>>>> Kudus to etcd team for making this blog and thanks for sharing.
>>>>>>=20
>>>>>> I feel like they're running a questionable configuration.
>>>>>>>>=20
>>>>>>> Looks like the test configuration
>>>>>> <https://github.com/coreos/dbtester/blob/89eb8d31addff1d9538
>>>>>> 235c20878a8637f24608c/agent/agent_zookeeper.go#L29>
>>>>>> does not have separate directory for transaction logs and =
snapshots
>> as
>>> it
>>>>>> does not have configuration for dataLogDir. So the configuration =
is
>> not
>>>>>> optimal. Would be interesting to see the numbers with updated
>>>>>> configuration.
>>>>>>=20
>>>>>> They mention that ZK snapshots "stop the world", and maybe I'm
>>> mistaken,
>>>>>>>> but
>>>>>>>>=20
>>>>>>> I didn't think that was right
>>>>>>=20
>>>>>> Right, ZK snapshots does not block processing pipeline as it is =
fuzzy
>>> and
>>>>>> it is done in a separate thread. The warning message "*To busy to
>> snap,
>>>>>> skipping*" mentioned in the blog is a sign that a snap shot is =
also
>>>>>> generating in progress, which could be caused by the write
>> contentions
>>>>>> created from serializing transaction logs that leads to longer =
than
>>>>>> expected snap shot generation. So "stop the world" is a side =
effect
>> of
>>>>>> resource contention, but not a design intention IMO.
>>>>>>=20
>>>>>> Also the blog mentions ZooKeeper as a key value store and I also =
want
>>> to
>>>>>> point out that ZooKeeper is more than a (metadata) key value =
store
>> has
>>>>>> features such as sessions, ephemerals, and watchers, and these =
design
>>>>>> choices were made I believe to make ZK more useful as a =
coordination
>>>>>> kernel, and these design choice also (negatively) contribute to =
the
>>>>>> performance and scalability of ZooKeeper.
>>>>>>=20
>>>>>>=20
>>>>>> On Tue, Feb 21, 2017 at 4:32 PM, Dan Benediktson <
>>>>>> dbenediktson@twitter.com.invalid> wrote:
>>>>>>=20
>>>>>> I kind of wonder about them only using one disk. I haven't
>> experimented
>>>>>>> with this in ZooKeeper, nor have I ever been a DBA, but with
>>> traditional
>>>>>>> database systems (which ZooKeeper should be basically identical =
to,
>> in
>>>>>>> this
>>>>>>> regard), it's a pretty common recommendation to put snapshots =
and
>>> TxLogs
>>>>>>> on
>>>>>>> different drives, for the obvious reason of avoiding one of the
>>> biggest
>>>>>>> problems laid out in that blog post: when snapshot happens, it
>>> contends
>>>>>>> with your log flushes, causing write latencies to explode. =
Suddenly
>>> you
>>>>>>> have tons more IO, and where it used to be nicely sequential, =
now
>> it's
>>>>>>> heavily randomized because of the two competing writers. It's =
kind
>> of
>>> the
>>>>>>> nature of benchmarks that there's always something you can =
nitpick,
>>> but
>>>>>>> still, I feel like they're running a questionable configuration.
>>>>>>>=20
>>>>>>> They mention that ZK snapshots "stop the world", and maybe I'm
>>> mistaken,
>>>>>>> but I didn't think that was right - I thought they were just =
slowing
>>>>>>> everything down because they write a lot and contend a lot. I'm
>> pretty
>>>>>>> sure
>>>>>>> ZK snapshots are fuzzy over a range of transactions, and
>> transactions
>>>>>>> keep
>>>>>>> applying during the snapshot, right?
>>>>>>>=20
>>>>>>> Thanks,
>>>>>>> Dan
>>>>>>>=20
>>>>>>> On Tue, Feb 21, 2017 at 2:24 PM, Benjamin Mahler <
>>> bmahler@mesosphere.io>
>>>>>>> wrote:
>>>>>>>=20
>>>>>>> I'm curious if folks here have seen the following write =
performance
>>>>>>>> comparison that was done by CoreOS on etc, Consul, and =
ZooKeeper:
>>>>>>>> https://coreos.com/blog/performance-of-etcd.html
>>>>>>>>=20
>>>>>>>> Sounds like performance comparison of reads and updates are =
coming
>>> next.
>>>>>>>> Are there any thoughts from folks here on this comparison so =
far?
>>>>>>>>=20
>>>>>>>> Thanks,
>>>>>>>> Ben
>>>>>>>>=20
>>>>>>>>=20
>>>>> --
>>>>> Alexander Binzberger
>>>>> System Designer - WINGcon AG
>>>>> Tel. +49 7543 966-119
>>>>>=20
>>>>> Sitz der Gesellschaft: Langenargen
>>>>> Registergericht: ULM, HRB 734260
>>>>> USt-Id.: DE232931635, WEEE-Id.: DE74015979
>>>>> Vorstand: thomasThomas Ehrle (Vorsitz), Fritz R. Paul
>> (Stellvertreter),
>>>>> Tobias Tre=C3=9F
>>>>> Aufsichtsrat: J=C3=BCrgen Maucher (Vorsitz), Andreas Paul =
(Stellvertreter),
>>>>> Martin Sauter
>>>>>=20
>>>>>=20
>>>=20
>>=20


--Apple-Mail=_DEF495AE-E72E-4A77-9520-A6E5B954F9B9--