Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 85E8411F7C for ; Wed, 9 Apr 2014 22:01:50 +0000 (UTC) Received: (qmail 25858 invoked by uid 500); 9 Apr 2014 22:01:49 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 25781 invoked by uid 500); 9 Apr 2014 22:01:49 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 25773 invoked by uid 99); 9 Apr 2014 22:01:49 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 22:01:49 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ptgoetz@gmail.com designates 209.85.216.50 as permitted sender) Received: from [209.85.216.50] (HELO mail-qa0-f50.google.com) (209.85.216.50) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 22:01:42 +0000 Received: by mail-qa0-f50.google.com with SMTP id ih12so3020128qab.23 for ; Wed, 09 Apr 2014 15:01:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:references:from:content-type:in-reply-to:message-id:date:to :content-transfer-encoding:mime-version; bh=9pCC2OL09kY2CTDuf1KfIvTMdurxBThiKqAtoNrtMvc=; b=rWYWwkOh8z1SAdq4VqEi83WE6gF7Kx6wVynNxSAOt7GLba6hx6Cr0VlAxR11KIhaUu 9SowMxWY9qCIDspEgQvmbLK+GpCr/ZLGUW4+stm2uwH/IskS601nKzty6MCKYldpABVq qOi5h6tZFCbfkAYGhhnMNvj6Ubr6aWQeD7YMBcdJ+U5HEqcWiPbX6CQzhAyy2qPxPm+K RryCowwqdCNG+PW1BVpHmLpIzwfwemKUaTacDBaQ5r5xgoio/24THe5pNYAq8mN9zgxw hCYn/8vJySZpE6WCeBwszx397qGNclsCpceZf2ji8hMzGBpZvQw8jlrouX9JpVMKByJy dguA== X-Received: by 10.140.82.167 with SMTP id h36mr15396088qgd.51.1397080880443; Wed, 09 Apr 2014 15:01:20 -0700 (PDT) Received: from [192.168.1.2] (pool-173-59-54-41.phlapa.fios.verizon.net. [173.59.54.41]) by mx.google.com with ESMTPSA id o11sm3957303qay.39.2014.04.09.15.01.18 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 09 Apr 2014 15:01:18 -0700 (PDT) Subject: Re: Topology is stuck References: From: "P. Taylor Goetz" Content-Type: multipart/alternative; boundary=Apple-Mail-4804746E-6D7A-4A9A-83A8-5AA66D40C8EA X-Mailer: iPad Mail (11D167) In-Reply-To: Message-Id: <01CF25B8-404D-4311-8A6C-55AC28B792D8@gmail.com> Date: Wed, 9 Apr 2014 18:01:17 -0400 To: "user@storm.incubator.apache.org" Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail-4804746E-6D7A-4A9A-83A8-5AA66D40C8EA Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Hey Jason, Do you know a way to reliably reproduce this? If so can you share the steps?= -Taylor > On Apr 9, 2014, at 5:52 PM, Jason Jackson wrote: >=20 > Fyi we're using Summingbird in production not Trident. However summingbird= does not give you exactly once semantics, it does give you a higher level o= f abstraction than Storm API though.=20 >=20 >=20 >> On Wed, Apr 9, 2014 at 2:50 PM, Jason Jackson wrote= : >> I have one theory that because reads in zookeeper are eventually consiste= nt, this is a necessary condition for the bug to manifest. So one way to tes= t this hypothesis is to run a zookeeper ensemble with 1 node, or a zookeeper= ensemble configured for 5 nodes, but take 2 of them offline, so that every w= rite operation only succeeds if every member of the ensemble sees the write.= This should produce strong consistent reads. If you run this test, let me k= now what the results are. (Clearly this isn't a good production system thoug= h as you're making a tradeoff for lower availability in favor of greater con= sistency, but the results could help narrow down the bug) >>=20 >>=20 >>> On Wed, Apr 9, 2014 at 2:43 PM, Jason Jackson wrot= e: >>> Yah it's probably a bug in trident. It would be amazing if someone figur= ed out the fix for this. I spent about 6 hours looking into, but couldn't fi= gure out why it was occuring.=20 >>>=20 >>> Beyond fixing this, one thing you could do to buy yourself time is disab= le batch retries in trident. There's no option for this in the API, but it's= like a 1 or 2 line change to the code. Obviously you loose exactly once sem= antics, but at least you would have a system that never falls behind real-ti= me.=20 >>>=20 >>>=20 >>>=20 >>>> On Wed, Apr 9, 2014 at 1:10 AM, Danijel Schiavuzzi wrote: >>>> Thanks Jason. However, I don't think that was the case in my stuck topo= logy, otherwise I'd have seen exceptions (thrown by my Trident functions) in= the worker logs. >>>>=20 >>>>=20 >>>>> On Wed, Apr 9, 2014 at 3:02 AM, Jason Jackson wr= ote: >>>>> An example of "corrupted input" that causes a batch to fail would be f= or example if you expected a schema to your data that you read off kafka, or= some queue, and for whatever reason the data didn't conform to your schema a= nd the function that you implement that you pass to stream.each throws an ex= ception when this unexpected situation occurs. This would cause the batch to= be retried, but it's deterministically failing, so the batch will be retrie= d forever.=20 >>>>>=20 >>>>>=20 >>>>>> On Mon, Apr 7, 2014 at 10:37 AM, Danijel Schiavuzzi wrote: >>>>>> Hi Jason, >>>>>>=20 >>>>>> Could you be more specific -- what do you mean by "corrupted input"? D= o you mean that there's a bug in Trident itself that causes the tuples in a b= atch to somehow become corrupted? >>>>>>=20 >>>>>> Thanks a lot! >>>>>>=20 >>>>>> Danijel >>>>>>=20 >>>>>>=20 >>>>>>> On Monday, April 7, 2014, Jason Jackson wrote:= >>>>>>> This could happen if you have corrupted input that always causes a b= atch to fail and be retried.=20 >>>>>>>=20 >>>>>>> I have seen this behaviour before and I didn't see corrupted input. I= t might be a bug in trident, I'm not sure. If you figure it out please updat= e this thread and/or submit a patch.=20 >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> On Mon, Mar 31, 2014 at 7:39 AM, Danijel Schiavuzzi wrote: >>>>>>> To (partially) answer my own question -- I still have no idea on the= cause of the stuck topology, but re-submitting the topology helps -- after r= e-submitting my topology is now running normally. >>>>>>>=20 >>>>>>>=20 >>>>>>> On Wed, Mar 26, 2014 at 6:04 PM, Danijel Schiavuzzi wrote: >>>>>>> Also, I did have multiple cases of my IBackingMap workers dying (bec= ause of RuntimeExceptions) but successfully restarting afterwards (I throw R= untimeExceptions in the BackingMap implementation as my strategy in rare SQL= database deadlock situations to force a worker restart and to fail+retry th= e batch). >>>>>>>=20 >>>>>>> =46rom the logs, one such IBackingMap worker death (and subsequent r= estart) resulted in the Kafka spout re-emitting the pending tuple: >>>>>>>=20 >>>>>>> 2014-03-22 16:26:43 s.k.t.TridentKafkaEmitter [INFO] re-emitting= batch, attempt 29698959:736 >>>>>>>=20 >>>>>>> This is of course the normal behavior of a transactional topology, b= ut this is the first time I've encountered a case of a batch retrying indefi= nitely. This is especially suspicious since the topology has been running fi= ne for 20 days straight, re-emitting batches and restarting IBackingMap work= ers quite a number of times. >>>>>>>=20 >>>>>>> I can see in my IBackingMap backing SQL database that the batch with= the exact txid value 29698959 has been committed -- but I suspect that coul= d come from another BackingMap, since there are two BackingMap instances run= ning (paralellismHint 2). >>>>>>>=20 >>>>>>> However, I have no idea why the batch is being retried indefinitely n= ow nor why it hasn't been successfully acked by Trident. >>>>>>>=20 >>>>>>> Any suggestions on the area (topology component) to focus my researc= h on? >>>>>>>=20 >>>>>>> Thanks, >>>>>>>=20 >>>>>>> On Wed, Mar 26, 2014 at 5:32 PM, Danijel Schiavuzzi wrote: >>>>>>> Hello, >>>>>>>=20 >>>>>>> I'm having problems with my transactional Trident topology. It has b= een running fine for about 20 days, and suddenly is stuck processing a singl= e batch, with no tuples being emitted nor tuples being persisted by the Trid= entState (IBackingMap). >>>>>>>=20 >>>>>>> It's a simple topology which consumes messages off a Kafka queue. Th= e spout is an instance of storm-kafka-0.8-plus TransactionalTridentKafkaSpou= t and I use the trident-mssql transactional TridentState implementation to p= ersistentAggregate() data into a SQL database. >>>>>>>=20 >>>>>>> In Zookeeper I can see Storm is re-trying a batch, i.e. >>>>>>>=20 >>>>>>> "/transactional//coordinator/currattempts" is "= {"29698959":6487}" >>>>>>>=20 >>>>>>> ... and the attempt count keeps increasing. It seems the batch with t= xid 29698959 is stuck, as the attempt count in Zookeeper keeps increasing --= seems like the batch isn't being acked by Trident and I have no idea why, e= specially since the topology has been running successfully the last 20 days.= >>>>>>>=20 >>>>>>> I did rebalance the topology on one occasion, after which it continu= ed running normally. Other than that, no other modifications were done. Stor= m is at version 0.9.0.1. >>>>>>>=20 >>>>>>> Any hints on how to debug the stuck topology? Any other useful info I= might provide? >>>>>>>=20 >>>>>>> Thanks, >>>>>>>=20 >>>>>>> --=20 >>>>>>> Danijel Schiavuzzi >>>>>>>=20 >>>>>>> E: danijel@schiavuzzi.com >>>>>>> W: www.schiavuzzi.com >>>>>>> T: +385989035562 >>>>>>> Skype: danijel.schiavuzzi >>>>>>>=20 >>>>>>>=20 >>>>>>>=20 >>>>>>> --=20 >>>>>>> Danijel Schiavuzzi >>>>>>>=20 >>>>>>> E: danije >>>>>>=20 >>>>>>=20 >>>>>> --=20 >>>>>> Danijel Schiavuzzi >>>>>>=20 >>>>>> E: danijel@schiavuzzi.com >>>>>> W: www.schiavuzzi.com >>>>>> T: +385989035562 >>>>>> Skype: danijels7 >>>>=20 >>>>=20 >>>>=20 >>>> --=20 >>>> Danijel Schiavuzzi >>>>=20 >>>> E: danijel@schiavuzzi.com >>>> W: www.schiavuzzi.com >>>> T: +385989035562 >>>> Skype: danijels7 >=20 --Apple-Mail-4804746E-6D7A-4A9A-83A8-5AA66D40C8EA Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit
Hey Jason,

Do you know a way to reliably reproduce this? If so can you share the steps?

-Taylor

On Apr 9, 2014, at 5:52 PM, Jason Jackson <jasonjckn@gmail.com> wrote:

Fyi we're using Summingbird in production not Trident. However summingbird does not give you exactly once semantics, it does give you a higher level of abstraction than Storm API though. 


On Wed, Apr 9, 2014 at 2:50 PM, Jason Jackson <jasonjckn@gmail.com> wrote:
I have one theory that because reads in zookeeper are eventually consistent, this is a necessary condition for the bug to manifest. So one way to test this hypothesis is to run a zookeeper ensemble with 1 node, or a zookeeper ensemble configured for 5 nodes, but take 2 of them offline, so that every write operation only succeeds if every member of the ensemble sees the write. This should produce strong consistent reads. If you run this test, let me know what the results are. (Clearly this isn't a good production system though as you're making a tradeoff for lower availability in favor of greater consistency, but the results could help narrow down the bug)


On Wed, Apr 9, 2014 at 2:43 PM, Jason Jackson <jasonjckn@gmail.com> wrote:
Yah it's probably a bug in trident. It would be amazing if someone figured out the fix for this. I spent about 6 hours looking into, but couldn't figure out why it was occuring. 

Beyond fixing this, one thing you could do to buy yourself time is disable batch retries in trident. There's no option for this in the API, but it's like a 1 or 2 line change to the code. Obviously you loose exactly once semantics, but at least you would have a system that never falls behind real-time. 



On Wed, Apr 9, 2014 at 1:10 AM, Danijel Schiavuzzi <danijel@schiavuzzi.com> wrote:
Thanks Jason. However, I don't think that was the case in my stuck topology, otherwise I'd have seen exceptions (thrown by my Trident functions) in the worker logs.


On Wed, Apr 9, 2014 at 3:02 AM, Jason Jackson <jasonjckn@gmail.com> wrote:
An example of "corrupted input" that causes a batch to fail would be for example if you expected a schema to your data that you read off kafka, or some queue, and for whatever reason the data didn't conform to your schema and the function that you implement that you pass to stream.each throws an exception when this unexpected situation occurs. This would cause the batch to be retried, but it's deterministically failing, so the batch will be retried forever. 


On Mon, Apr 7, 2014 at 10:37 AM, Danijel Schiavuzzi <danijel@schiavuzzi.com> wrote:
Hi Jason,

Could you be more specific -- what do you mean by "corrupted input"? Do you mean that there's a bug in Trident itself that causes the tuples in a batch to somehow become corrupted?

Thanks a lot!

Danijel


On Monday, April 7, 2014, Jason Jackson <jasonjckn@gmail.com> wrote:
This could happen if you have corrupted input that always causes a batch to fail and be retried. 

I have seen this behaviour before and I didn't see corrupted input. It might be a bug in trident, I'm not sure. If you figure it out please update this thread and/or submit a patch. 



On Mon, Mar 31, 2014 at 7:39 AM, Danijel Schiavuzzi <danijel@schiavuzzi.com> wrote:
To (partially) answer my own question -- I still have no idea on the cause of the stuck topology, but re-submitting the topology helps -- after re-submitting my topology is now running normally.


On Wed, Mar 26, 2014 at 6:04 PM, Danijel Schiavuzzi <danijel@schiavuzzi.com> wrote:
Also, I did have multiple cases of my IBackingMap workers dying (because of RuntimeExceptions) but successfully restarting afterwards (I throw RuntimeExceptions in the BackingMap implementation as my strategy in rare SQL database deadlock situations to force a worker restart and to fail+retry the batch).

From the logs, one such IBackingMap worker death (and subsequent restart) resulted in the Kafka spout re-emitting the pending tuple:

    2014-03-22 16:26:43 s.k.t.TridentKafkaEmitter [INFO] re-emitting batch, attempt 29698959:736

This is of course the normal behavior of a transactional topology, but this is the first time I've encountered a case of a batch retrying indefinitely. This is especially suspicious since the topology has been running fine for 20 days straight, re-emitting batches and restarting IBackingMap workers quite a number of times.

I can see in my IBackingMap backing SQL database that the batch with the exact txid value 29698959 has been committed -- but I suspect that could come from another BackingMap, since there are two BackingMap instances running (paralellismHint 2).

However, I have no idea why the batch is being retried indefinitely now nor why it hasn't been successfully acked by Trident.

Any suggestions on the area (topology component) to focus my research on?

Thanks,

On Wed, Mar 26, 2014 at 5:32 PM, Danijel Schiavuzzi <danijel@schiavuzzi.com> wrote:
Hello,

I'm having problems with my transactional Trident topology. It has been running fine for about 20 days, and suddenly is stuck processing a single batch, with no tuples being emitted nor tuples being persisted by the TridentState (IBackingMap).

It's a simple topology which consumes messages off a Kafka queue. The spout is an instance of storm-kafka-0.8-plus TransactionalTridentKafkaSpout and I use the trident-mssql transactional TridentState implementation to persistentAggregate() data into a SQL database.

In Zookeeper I can see Storm is re-trying a batch, i.e.

     "/transactional/<myTopologyName>/coordinator/currattempts" is "{"29698959":6487}"

... and the attempt count keeps increasing. It seems the batch with txid 29698959 is stuck, as the attempt count in Zookeeper keeps increasing -- seems like the batch isn't being acked by Trident and I have no idea why, especially since the topology has been running successfully the last 20 days.

I did rebalance the topology on one occasion, after which it continued running normally. Other than that, no other modifications were done. Storm is at version 0.9.0.1.

Any hints on how to debug the stuck topology? Any other useful info I might provide?

Thanks,

--
Danijel Schiavuzzi

E: danijel@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijel.schiavuzzi



--
Danijel Schiavuzzi

E: danije


--
Danijel Schiavuzzi

E: danijel@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7




--
Danijel Schiavuzzi

E: danijel@schiavuzzi.com
W: www.schiavuzzi.com
T: +385989035562
Skype: danijels7



--Apple-Mail-4804746E-6D7A-4A9A-83A8-5AA66D40C8EA--