Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0D607115B4 for ; Sun, 6 Apr 2014 21:59:56 +0000 (UTC) Received: (qmail 94505 invoked by uid 500); 6 Apr 2014 21:59:56 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 94436 invoked by uid 500); 6 Apr 2014 21:59:55 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 94426 invoked by uid 99); 6 Apr 2014 21:59:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Apr 2014 21:59:55 +0000 X-ASF-Spam-Status: No, hits=2.1 required=5.0 tests=HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jasonjckn@gmail.com designates 209.85.216.48 as permitted sender) Received: from [209.85.216.48] (HELO mail-qa0-f48.google.com) (209.85.216.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 06 Apr 2014 21:59:51 +0000 Received: by mail-qa0-f48.google.com with SMTP id dc16so1024902qab.21 for ; Sun, 06 Apr 2014 14:59:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=aDgt/6Tu2T0MrechH7XKN0GbypGp1lmO6uZA3gFOHPA=; b=ptL5pCjJzPjkYxvyjoAHa8XAAuYSdWd8mTGMcgeT7UvaJwQHDIi5Zm6I6uSr3plhf8 +co92Fe6Km7qbuJPGzDJ0SPP3GN377o5PcdwX5VbyaN2YWBm2D/bkNR6jcw8ux/9aC8T e71zmcgDJgyrA8oEyA9d0EcEcWgKHZZpbiZrLwhOxzdNhdPMPi3N3oKccia8mzORWvzU rTi5JtcdQL9pkxVMcyiTtj95Z0R4ibD/rFmXaieihaTjjzSbxg18/iou7axPLnh7VFJG YrHrTJ1KM904gwC0qr3xkAbCxB9Px56yKk3YhK8+ujwh/UCtxe7RjLcz4etW5qCfhgD4 5XUw== X-Received: by 10.224.36.194 with SMTP id u2mr10823720qad.73.1396821569014; Sun, 06 Apr 2014 14:59:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.96.151.167 with HTTP; Sun, 6 Apr 2014 14:59:08 -0700 (PDT) In-Reply-To: References: From: Jason Jackson Date: Sun, 6 Apr 2014 14:59:08 -0700 Message-ID: Subject: Re: ACK performance hit & Loggly abandoning Storm To: user Content-Type: multipart/alternative; boundary=001a11c2a2406f915504f666dd99 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2a2406f915504f666dd99 Content-Type: text/plain; charset=ISO-8859-1 It would of been far more useful if they measured the systems in terms of dollars, as each system makes different tradeoffs. Certainly when you enable acking you may become bottlenecked on CPU at that point instead of being bottlenecked on disk/kafka. So one thing you can do is move to hardware with higher class CPUs to solve the bottleneck. The system they built is persisting intermediary queues between components in a topology. So while this will reduce CPU load by not needing an acking system, you will need more disks as potentially any of the intermediately queues can start to fill up now, you need to reserve capacity for worst case scenario. Potentially in terms of dollars the tradeoff to use more disks has marginally better total cost. On Fri, Apr 4, 2014 at 6:55 PM, Benjamin Black wrote: > No part of the post made any sense to me. There is a significant > performance hit when moving to reliable operation in any system and Storm > is clearly doing a good job if a custom built solution can only manage 25% > more throughput. > > > On Fri, Apr 4, 2014 at 4:10 PM, Neelesh wrote: > >> Its an interesting read. The blog is vague on some details - with ACK on, >> the throughput was 80K/s. With their custom solution its 100K/s. Assuming >> they were both deployed on similar hardware (I do not know , the blog does >> not confirm either way), the difference is not something that warrants a >> custom framework to me. Obviously its working better for Loggly. >> >> >> On Fri, Apr 4, 2014 at 8:26 AM, Otis Gospodnetic < >> otis.gospodnetic@gmail.com> wrote: >> >>> Hi, >>> >>> Apparently Loggly decided to ditch Storm when they got hit by the 2.5x >>> performance degradation factor after turning on ACKing: >>> https://www.loggly.com/what-we-learned-about-scaling-with-apache-storm/ >>> >>> How does one minimize this performance hit? >>> Or maybe newer versions of Storm perform better with ACK? (Loggly tested >>> 0.82, they say) >>> >>> Thanks, >>> Otis >>> -- >>> Performance Monitoring * Log Analytics * Search Analytics >>> Solr & Elasticsearch Support * http://sematext.com/ >>> >>> >> > --001a11c2a2406f915504f666dd99 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
It would of been far more useful if they measured the syst= ems in terms of dollars, as each system makes different tradeoffs. Certainl= y when you enable acking you may become bottlenecked on CPU at that point i= nstead of being bottlenecked on disk/kafka. So one thing you can do is move= to hardware with higher class CPUs to solve the bottleneck. The system the= y built is persisting intermediary queues between components in a topology.= So while this will reduce CPU load by not needing an acking system, you wi= ll need more disks as potentially any of the intermediately queues can star= t to fill up now, you need to reserve capacity for worst case scenario. Pot= entially in terms of dollars the tradeoff to use more disks has marginally = better total cost.=A0




On Fri, Apr 4, 2014 at 6:55 PM, Benjamin Black <b@b3k.us>= wrote:
No part of the post made an= y sense to me. There is a significant performance hit when moving to reliab= le operation in any system and Storm is clearly doing a good job if a custo= m built solution can only manage 25% more throughput.


On Fri, Apr 4= , 2014 at 4:10 PM, Neelesh <neeleshs@gmail.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Its an interesting read. The blog is vague on some details= - with ACK on, the throughput was 80K/s. With their custom solution its 10= 0K/s. Assuming they were both deployed on similar hardware (I do not know ,= the blog does not confirm either way), the difference is not something tha= t warrants a custom framework to me. Obviously its working better for Loggl= y.=A0


On Fri, Apr 4= , 2014 at 8:26 AM, Otis Gospodnetic <otis.gospodnetic@gmail.com> wrote:
Hi,

Appa= rently Loggly decided to ditch Storm when they got hit by the 2.5x performa= nce degradation factor after turning on ACKing:

How does one minimize this performance hit?
=
Or maybe newer versions of Storm perform better with ACK? (Loggly test= ed 0.82, they say)

Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
= Solr & Elasticsearch Support * http://sematext.com/




--001a11c2a2406f915504f666dd99--