Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A13E5200B7E for ; Tue, 6 Sep 2016 15:45:10 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 9FDD2160ABF; Tue, 6 Sep 2016 13:45:10 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C266D160AAD for ; Tue, 6 Sep 2016 15:45:09 +0200 (CEST) Received: (qmail 61652 invoked by uid 500); 6 Sep 2016 13:45:08 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 61642 invoked by uid 99); 6 Sep 2016 13:45:08 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Sep 2016 13:45:08 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 3AD21C03BC for ; Tue, 6 Sep 2016 13:45:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.179 X-Spam-Level: * X-Spam-Status: No, score=1.179 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 5V2HgYHFv2mr for ; Tue, 6 Sep 2016 13:45:04 +0000 (UTC) Received: from mail-lf0-f50.google.com (mail-lf0-f50.google.com [209.85.215.50]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 258645F479 for ; Tue, 6 Sep 2016 13:45:04 +0000 (UTC) Received: by mail-lf0-f50.google.com with SMTP id u14so23873246lfd.1 for ; Tue, 06 Sep 2016 06:45:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=6HzlobWR3e1Gj4ohNPrHI5lW1LyeGQch6ZSMjgJ/umc=; b=abl1gM40yuiydvyMdOxmpm8oFILj3pXx0fp54Bmf3Ji4wvuIMeJppBLqho1M9Sku6r MWlybQF5zyBWL+Bs37hJE3NUnvRpGLX9nx3FdqigL/sUBK1h1/poZnsa37c+CMBzXhlF yF2NNreyNzUqTOpUAlVd6Yyp6e+CbGgpEXVf8632aoT+xYGRfTIIiwJfnQrXWRJU4m/a 7NPN3IqH5ibR549idxwtEZup07bkxdjcdEyM9gDZzUhiVRM9R46D6VwVAXuhwP7K2li4 bgJ1+aQa0gXSlRCxWkC7ly/zhbxx4oDwAuLc6TXwHgkS/bIII3TGKzupxhLAOVU+v/i+ B3xA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=6HzlobWR3e1Gj4ohNPrHI5lW1LyeGQch6ZSMjgJ/umc=; b=MAJfx1Le70KA2jtj5Jxw9aaQUKbavwTl9f0Z33A8GO7QYMY3Ut+leV3BwJaaudWWkj NJVl9e+uPOWgtvN76dy0g99p46I4q1XhWqTLSoTobL3s+E0UrBJjxNRlnOxeHmz5fjEi awfU4fCUlM0TRWPAd1YqtPT7kMvEa53t3rVfXWW8YtHSmdO85b9Z8mafus2f+gG8Nx/C M2sOMPpwaUg5R8IJuvuUl0GL1W3ktyff1QnK/Z5dNAvwUqwIPkCf4mh9mp3o3vG/GZx9 WRL2kJcnS+aNzSM/Lwpl49RN5rO1IQYH9IxnSKPa2ty6jUV4LIVmMG/vxahJ8I7qWxsQ exPA== X-Gm-Message-State: AE9vXwMLRxB3rwLNBZmYJ3ZNCslnvKjtfFTC3nRd3oaO4iBumFdbk5CVju8+WNUt2yV7TFCHztIlHc8WLFRXvw== X-Received: by 10.25.168.144 with SMTP id r138mr12835192lfe.136.1473169502251; Tue, 06 Sep 2016 06:45:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.25.20.207 with HTTP; Tue, 6 Sep 2016 06:45:01 -0700 (PDT) In-Reply-To: References: From: Kristopher Kane Date: Tue, 6 Sep 2016 09:45:01 -0400 Message-ID: Subject: Re: Increasing worker parallelism decreases throughput and increases tuple timeout To: user@storm.apache.org Content-Type: multipart/alternative; boundary=001a11411310e02fc6053bd7005e archived-at: Tue, 06 Sep 2016 13:45:10 -0000 --001a11411310e02fc6053bd7005e Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Come to think of it, I did see RestUtils rank some what higher in the visualvm CPU profiler but did not give it the attention it deserved. On Tue, Sep 6, 2016 at 9:39 AM, Aaron Niskod=C3=A9-Dossett wrote: > Hi Kris, > > One possibility is that the Serializer isn't actually caching the schema > <-> id mappings and is hitting the schema registry every time. The call = to > register() in getFingerprint() [1] in particular can be a finicky since t= he > cache is ultimately in an IDENTITY hash map, not a regular old hashmap[2]= . > I'm familiar with the Avro deserializer you're using and though it > accounted for this, but perhaps not. > > You could add timing information to the getFingerprint() and getSchema() > calls in ConfluentAvroSerializer. If the results indicate cache misses, > that's probably your culprit. > > Best, Aaron > > > [1] https://github.com/apache/storm/blob/master/external/ > storm-hdfs/src/main/java/org/apache/storm/hdfs/avro/ > ConfluentAvroSerializer.java#L66 > [2] https://github.com/confluentinc/schema-registry/ > blob/v1.0/client/src/main/java/io/confluent/kafka/schemaregistry/client/ > CachedSchemaRegistryClient.java#L79 > > On Tue, Sep 6, 2016 at 7:40 AM Kristopher Kane > wrote: > >> Hi everyone. >> >> I have a simple topology that uses the Avro serializer ( >> https://github.com/apache/storm/blob/master/external/ >> storm-hdfs/src/main/java/org/apache/storm/hdfs/avro/ >> ConfluentAvroSerializer.java) and writes to Elasticsearch. >> >> The topology is like this: >> >> Kafka (raw scheme) -> Avro deserializer -> Elasticsearch >> >> This topology runs well with one worker, however, once I add one more >> worker (total of two) and change nothing else, the topology throughput >> drops and tuples start timing out. >> >> I've attached visualvm/jstatd to the workers when in multi worker mode - >> and added some jmx configs to the worker opts - but I am unable to see >> anything glaring. >> >> I've never seen Storm act this way but have also never worked with a >> custom serializer so assume that it is the culprit but I cannot explain >> why. >> >> Any pointers would be appreciated. >> >> Kris >> >> >> >> >> --001a11411310e02fc6053bd7005e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Come to think of it, I did see RestUtils rank some what hi= gher in the visualvm CPU profiler but did not give it the attention it dese= rved.=C2=A0

= On Tue, Sep 6, 2016 at 9:39 AM, Aaron Niskod=C3=A9-Dossett <dossett@gmail.= com> wrote:
Hi Kris,

One possibility is that the Serializer isn= 9;t actually caching the schema <-> id mappings and is hitting the sc= hema registry every time.=C2=A0 The call to register() in getFingerprint() = [1] in particular can be a finicky since the cache is ultimately in an IDEN= TITY hash map, not a regular old hashmap[2].=C2=A0 I'm familiar with th= e Avro deserializer you're using and though it accounted for this, but = perhaps not.

You could add timing information to t= he getFingerprint() and getSchema() calls in ConfluentAvroSerializer.=C2=A0= If the results indicate cache misses, that's probably your culprit.

Best, Aaron


<= div class=3D"HOEnZb">

On Tue, Sep 6, 2016 at 7:40 AM Kristopher Kane <kkane.list@gmail.com> = wrote:
Hi everyone= . =C2=A0

I have a simple topology that uses the Avro ser= ializer (https://github.com/apache/storm/blob/master/ext= ernal/storm-hdfs/src/main/java/org/apache/storm/hdfs/avro/Co= nfluentAvroSerializer.java)=C2=A0 and writes to Elasticsearch.=C2=A0

The topology is like this:

= Kafka (raw scheme) -> Avro deserializer -> Elasticsearch
This topology runs well with one worker, however, once I add o= ne more worker (total of two) and change nothing else, the topology through= put drops and tuples start timing out. =C2=A0

I= 9;ve attached visualvm/jstatd to the workers when in multi worker mode - an= d added some jmx configs to the worker opts - but I am unable to see anythi= ng glaring.

I've never seen Storm act this way= but have also never worked with a custom serializer so assume that it is t= he culprit but I cannot explain why.=C2=A0

Any poi= nters would be appreciated.=C2=A0

Kris
<= br>




--001a11411310e02fc6053bd7005e--