zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bobby Evans <ev...@yahoo-inc.com.INVALID>
Subject Re: Paper
Date Thu, 26 Jan 2017 21:14:45 GMT
We did think about ram disks a little, but the plan still is to have the source code, in one
form or another, morph into a small distributed time series database for the metrics.  With
that in mind we thought it would be better to take a step in that direction.  Yes a ram disk
would likely have provided similar performance.  Although we would have still wanted to separate
out just the metrics to the ram disk backed ZK, because we store other more critical data
in ZK too, that need more durability guarantees.

- Bobby

On Thursday, January 26, 2017, 10:36:01 AM CST, Patrick Hunt <phunt@apache.org> wrote:Very
interesting results and real world insights. Thanks for
creating/sharing.

One thing I noticed is that you mentioned considering SSDs, had you also
considered using ram disks? I've seen some scenarios where that has been
very successful.

Patrick

On Thu, Jan 26, 2017 at 6:28 AM, Bobby Evans <evans@yahoo-inc.com.invalid>
wrote:

> As one of the authors of pacemaker in Apache Storm (and the paper), I am
> happy to answer any questions about why we did it or how it works.  The
> reality of it is storm was, and still is by default, abusing zookeeper by
> trying to store a massive amount of metrics in it, instead of the
> configuration/coordination it was designed for. And since storm metrics
> don't really need strong consistency or even that much in terms of
> reliability guarantees we stood up a netty server in front of a
> ConcurrentHashMap (quite literately) and then wrote a client that could
> handle fail-over.
> It really is meant as a scalability stepping stone until we can get to the
> point that all the metrics go to a TSDB that is actually designed for
> metrics. But like I said if you have any questions I am happy to answer
> them.
> Sadly because of the way IEEE works neither I nor my employer own the copy
> right to that paper any more so I can't even put a copy of it up for you to
> read.
>
>
> - Bobby
>
> On Thursday, January 26, 2017, 6:44:56 AM CST, ibrahim El-sanosi <
> ibrahimsabattt@gmail.com> wrote:Hi folk,
>
> There is a paper published recently "PaceMaker: When ZooKeeper Arteries Get
> Clogged in Storm Clusters" [1]. It may worth to read.
>
> [1]
> http://ieeexplore.ieee.org/document/7820303/?tp=&
> arnumber=7820303&contentType=Conference%20Publications&dld=
> eWFob28uY29t&source=SEARCHALERT
>
> Ibrahim
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message