storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Larry Palmer <larry.a.pal...@gmail.com>
Subject Storm throughput questions
Date Wed, 05 Feb 2014 17:46:22 GMT
Hi,
I'm evaluating Storm for use in a fairly high throughput environment: ~100
billion messages/day on internal servers.  I've seen rates for Storm quoted
of around a million tuples/second/server, but I wonder if anyone could help
clarify the precise definition of that metric.

For discussion's sake, assume a rather trivial network of a single spout
S1, feeding a single bolt B1, which sends each input tuple to two
downstream bolts: B2 and B3. In that case, if the cluster were running at
"1,000,000 tuples/second", what would the input rate be?

   1. Does it mean a million tuples per second coming into S1?
   2. Or maybe each input tuple is considered to be three tuples: one from
   S1 to B1, and one each from B1->B2 and B1->B3, giving an input rate of
   333,000 tuples/second?
   3. Are the acks considered to be tuples as well, giving maybe 6 tuples
   for every input tuple and an effective input rate of 166,000 tuples/second?
   4. Are there perhaps even more tuples factoring into the mix (metrics
   etc) lowering the input rate even more?

Having established the definition of tuples/second, is a million/server
still the current benchmark or might it be a bit higher these days due to
hardware improvements? Is it safe to assume that a million tuples would be
with bolts that were actually doing some sort of modest computation, or is
that just the barebones getting tuples in and out as fast as possible doing
nothing with them? What kind of numbers are people seeing per server in
practical implementations?

Also, has anyone done any experimentation with Storm running with
hyperthreading on vs off? Our experience with Hadoop has shown
hyperthreading to either help performance very little or hurt slightly -
would the same be expected for Storm?

Thanks!

Mime
View raw message