Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EC06111F2E for ; Wed, 9 Apr 2014 21:53:00 +0000 (UTC) Received: (qmail 5339 invoked by uid 500); 9 Apr 2014 21:53:00 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 5302 invoked by uid 500); 9 Apr 2014 21:53:00 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 5293 invoked by uid 99); 9 Apr 2014 21:52:59 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 21:52:59 +0000 X-ASF-Spam-Status: No, hits=2.1 required=5.0 tests=HK_RANDOM_ENVFROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jasonjckn@gmail.com designates 209.85.216.44 as permitted sender) Received: from [209.85.216.44] (HELO mail-qa0-f44.google.com) (209.85.216.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Apr 2014 21:52:55 +0000 Received: by mail-qa0-f44.google.com with SMTP id hw13so3103596qab.31 for ; Wed, 09 Apr 2014 14:52:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=pPGSEjrHTaUe8Zw8SF1d4J3zGEhsoUEz0Yf5R17B2S0=; b=ug9Xg0+UR0nH3zLuIeS6B8zU09nRzpsjue8kg8+G7OreeQPlVW/mzpgaJGRHBW0LQR yziFe1xu0gqtOF7lSzZ2FIRnvvHaQI8Ip0Rf9cXExKBfQ+YI8X0y0AGS/ELILtxoSBXR bRJZC89bHmGodSwh8xGg143npV3izsSWsWEWtDsdUXWome8O8GUti96p38SwEP6Sdvm+ XxK9iQYIfXWHwd5lopddQwycfIFMNm4pc+vUukoPIn3RJ54XjZ632qfmvwFKrGYLvh3I 4altAraDzeTe65lOGDnJ0LmeX0vmpJTahsi2CbenJn9QpSbZPhs8+IUEYsLVsALCe/in gAuA== X-Received: by 10.140.19.68 with SMTP id 62mr15463935qgg.55.1397080354605; Wed, 09 Apr 2014 14:52:34 -0700 (PDT) MIME-Version: 1.0 Received: by 10.96.151.167 with HTTP; Wed, 9 Apr 2014 14:52:14 -0700 (PDT) In-Reply-To: References: From: Jason Jackson Date: Wed, 9 Apr 2014 14:52:14 -0700 Message-ID: Subject: Re: Topology is stuck To: user Content-Type: multipart/alternative; boundary=001a11353c74426d8804f6a31ec6 X-Virus-Checked: Checked by ClamAV on apache.org --001a11353c74426d8804f6a31ec6 Content-Type: text/plain; charset=ISO-8859-1 Fyi we're using Summingbird in production not Trident. However summingbird does not give you exactly once semantics, it does give you a higher level of abstraction than Storm API though. On Wed, Apr 9, 2014 at 2:50 PM, Jason Jackson wrote: > I have one theory that because reads in zookeeper are eventually > consistent, this is a necessary condition for the bug to manifest. So one > way to test this hypothesis is to run a zookeeper ensemble with 1 node, or > a zookeeper ensemble configured for 5 nodes, but take 2 of them offline, so > that every write operation only succeeds if every member of the ensemble > sees the write. This should produce strong consistent reads. If you run > this test, let me know what the results are. (Clearly this isn't a good > production system though as you're making a tradeoff for lower availability > in favor of greater consistency, but the results could help narrow down the > bug) > > > On Wed, Apr 9, 2014 at 2:43 PM, Jason Jackson wrote: > >> Yah it's probably a bug in trident. It would be amazing if someone >> figured out the fix for this. I spent about 6 hours looking into, but >> couldn't figure out why it was occuring. >> >> Beyond fixing this, one thing you could do to buy yourself time is >> disable batch retries in trident. There's no option for this in the API, >> but it's like a 1 or 2 line change to the code. Obviously you loose exactly >> once semantics, but at least you would have a system that never falls >> behind real-time. >> >> >> >> On Wed, Apr 9, 2014 at 1:10 AM, Danijel Schiavuzzi < >> danijel@schiavuzzi.com> wrote: >> >>> Thanks Jason. However, I don't think that was the case in my stuck >>> topology, otherwise I'd have seen exceptions (thrown by my Trident >>> functions) in the worker logs. >>> >>> >>> On Wed, Apr 9, 2014 at 3:02 AM, Jason Jackson wrote: >>> >>>> An example of "corrupted input" that causes a batch to fail would be >>>> for example if you expected a schema to your data that you read off kafka, >>>> or some queue, and for whatever reason the data didn't conform to your >>>> schema and the function that you implement that you pass to stream.each >>>> throws an exception when this unexpected situation occurs. This would cause >>>> the batch to be retried, but it's deterministically failing, so the batch >>>> will be retried forever. >>>> >>>> >>>> On Mon, Apr 7, 2014 at 10:37 AM, Danijel Schiavuzzi < >>>> danijel@schiavuzzi.com> wrote: >>>> >>>>> Hi Jason, >>>>> >>>>> Could you be more specific -- what do you mean by "corrupted input"? >>>>> Do you mean that there's a bug in Trident itself that causes the tuples in >>>>> a batch to somehow become corrupted? >>>>> >>>>> Thanks a lot! >>>>> >>>>> Danijel >>>>> >>>>> >>>>> On Monday, April 7, 2014, Jason Jackson wrote: >>>>> >>>>>> This could happen if you have corrupted input that always causes a >>>>>> batch to fail and be retried. >>>>>> >>>>>> I have seen this behaviour before and I didn't see corrupted input. >>>>>> It might be a bug in trident, I'm not sure. If you figure it out please >>>>>> update this thread and/or submit a patch. >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Mar 31, 2014 at 7:39 AM, Danijel Schiavuzzi < >>>>>> danijel@schiavuzzi.com> wrote: >>>>>> >>>>>> To (partially) answer my own question -- I still have no idea on the >>>>>> cause of the stuck topology, but re-submitting the topology helps -- after >>>>>> re-submitting my topology is now running normally. >>>>>> >>>>>> >>>>>> On Wed, Mar 26, 2014 at 6:04 PM, Danijel Schiavuzzi < >>>>>> danijel@schiavuzzi.com> wrote: >>>>>> >>>>>> Also, I did have multiple cases of my IBackingMap workers dying >>>>>> (because of RuntimeExceptions) but successfully restarting afterwards (I >>>>>> throw RuntimeExceptions in the BackingMap implementation as my strategy in >>>>>> rare SQL database deadlock situations to force a worker restart and to >>>>>> fail+retry the batch). >>>>>> >>>>>> From the logs, one such IBackingMap worker death (and subsequent >>>>>> restart) resulted in the Kafka spout re-emitting the pending tuple: >>>>>> >>>>>> 2014-03-22 16:26:43 s.k.t.TridentKafkaEmitter [INFO] re-emitting >>>>>> batch, attempt 29698959:736 >>>>>> >>>>>> This is of course the normal behavior of a transactional topology, >>>>>> but this is the first time I've encountered a case of a batch retrying >>>>>> indefinitely. This is especially suspicious since the topology has been >>>>>> running fine for 20 days straight, re-emitting batches and restarting >>>>>> IBackingMap workers quite a number of times. >>>>>> >>>>>> I can see in my IBackingMap backing SQL database that the batch with >>>>>> the exact txid value 29698959 has been committed -- but I suspect that >>>>>> could come from another BackingMap, since there are two BackingMap >>>>>> instances running (paralellismHint 2). >>>>>> >>>>>> However, I have no idea why the batch is being retried indefinitely >>>>>> now nor why it hasn't been successfully acked by Trident. >>>>>> >>>>>> Any suggestions on the area (topology component) to focus my research >>>>>> on? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> On Wed, Mar 26, 2014 at 5:32 PM, Danijel Schiavuzzi < >>>>>> danijel@schiavuzzi.com> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> I'm having problems with my transactional Trident topology. It has >>>>>> been running fine for about 20 days, and suddenly is stuck processing a >>>>>> single batch, with no tuples being emitted nor tuples being persisted by >>>>>> the TridentState (IBackingMap). >>>>>> >>>>>> It's a simple topology which consumes messages off a Kafka queue. The >>>>>> spout is an instance of storm-kafka-0.8-plus TransactionalTridentKafkaSpout >>>>>> and I use the trident-mssql transactional TridentState implementation to >>>>>> persistentAggregate() data into a SQL database. >>>>>> >>>>>> In Zookeeper I can see Storm is re-trying a batch, i.e. >>>>>> >>>>>> "/transactional//coordinator/currattempts" is >>>>>> "{"29698959":6487}" >>>>>> >>>>>> ... and the attempt count keeps increasing. It seems the batch with >>>>>> txid 29698959 is stuck, as the attempt count in Zookeeper keeps increasing >>>>>> -- seems like the batch isn't being acked by Trident and I have no idea >>>>>> why, especially since the topology has been running successfully the last >>>>>> 20 days. >>>>>> >>>>>> I did rebalance the topology on one occasion, after which it >>>>>> continued running normally. Other than that, no other modifications were >>>>>> done. Storm is at version 0.9.0.1. >>>>>> >>>>>> Any hints on how to debug the stuck topology? Any other useful info I >>>>>> might provide? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -- >>>>>> Danijel Schiavuzzi >>>>>> >>>>>> E: danijel@schiavuzzi.com >>>>>> W: www.schiavuzzi.com >>>>>> T: +385989035562 >>>>>> Skype: danijel.schiavuzzi >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Danijel Schiavuzzi >>>>>> >>>>>> E: danije >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Danijel Schiavuzzi >>>>> >>>>> E: danijel@schiavuzzi.com >>>>> W: www.schiavuzzi.com >>>>> T: +385989035562 >>>>> Skype: danijels7 >>>>> >>>> >>>> >>> >>> >>> -- >>> Danijel Schiavuzzi >>> >>> E: danijel@schiavuzzi.com >>> W: www.schiavuzzi.com >>> T: +385989035562 >>> Skype: danijels7 >>> >> >> > --001a11353c74426d8804f6a31ec6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Fyi we're using Summingbird in production not Trident.= However summingbird does not give you exactly once semantics, it does give= you a higher level of abstraction than Storm API though.=A0


On Wed, Apr 9, 2014 at 2:50 PM, Jason Ja= ckson <jasonjckn@gmail.com> wrote:
I have one theory that because reads in zookeeper are even= tually consistent, this is a necessary condition for the bug to manifest. S= o one way to test this hypothesis is to run a zookeeper ensemble with 1 nod= e, or a zookeeper ensemble configured for 5 nodes, but take 2 of them offli= ne, so that every write operation only succeeds if every member of the ense= mble sees the write. This should produce strong consistent reads. If you ru= n this test, let me know what the results are. (Clearly this isn't a go= od production system though as you're making a tradeoff for lower avail= ability in favor of greater consistency, but the results could help narrow = down the bug)


On Wed, Apr 9= , 2014 at 2:43 PM, Jason Jackson <jasonjckn@gmail.com> wro= te:
Yah it's probably a bug= in trident. It would be amazing if someone figured out the fix for this. I= spent about 6 hours looking into, but couldn't figure out why it was o= ccuring.=A0

Beyond fixing this, one thing you could do to buy yourself t= ime is disable batch retries in trident. There's no option for this in = the API, but it's like a 1 or 2 line change to the code. Obviously you = loose exactly once semantics, but at least you would have a system that nev= er falls behind real-time.=A0



On Wed, Apr 9, 2014 at 1:10 AM, Danijel = Schiavuzzi <danijel@schiavuzzi.com> wrote:
Thanks Jason. However, I don't think that was the case= in my stuck topology, otherwise I'd have seen exceptions (thrown by my= Trident functions) in the worker logs.


On Wed, Apr 9, 2014 at 3:02 AM, Jason Ja= ckson <jasonjckn@gmail.com> wrote:
An example of "corrupted input" that causes a ba= tch to fail would be for example if you expected a schema to your data that= you read off kafka, or some queue, and for whatever reason the data didn&#= 39;t conform to your schema and the function that you implement that you pa= ss to stream.each throws an exception when this unexpected situation occurs= . This would cause the batch to be retried, but it's deterministically = failing, so the batch will be retried forever.=A0


On Mon, Apr 7= , 2014 at 10:37 AM, Danijel Schiavuzzi <danijel@schiavuzzi.com>= ; wrote:
Hi Jason,

Could you be mo= re=A0specific --=A0what do you mean by "corrupted input"? Do=A0yo= u mean that=A0there's=A0a bug in Trident itself=A0that causes the tuple= s in a batch=A0to somehow=A0become corrupted?

Thanks a lot!
<= br>
Danijel


On Monday, April 7, 2014, Jason Jackson <jasonjckn@gmail.com&g= t; wrote:
This could happen if you have corrupted input that always = causes a batch to fail and be retried.=A0

I have seen th= is behaviour before and I didn't see corrupted input. It might be a bug= in trident, I'm not sure. If you figure it out please update this thre= ad and/or submit a patch.=A0



On Mon, Mar 31, 2014 at 7:39 AM, Dan= ijel Schiavuzzi <danijel@schiavuzzi.com> wrote:
To (partially) answer my own question -- I still = have no idea on the cause of the stuck topology, but re-submitting the topo= logy helps -- after re-submitting my topology is now running normally.


On Wed, Mar 26, 2014 at 6:04 PM, Danijel Schiavuzzi <danijel@schiavuzzi.com> wrote:
Also, I did have multiple cases of my IBackingM= ap workers dying (because of RuntimeExceptions) but successfully restarting= afterwards (I throw RuntimeExceptions in the BackingMap implementation as = my strategy in rare SQL database deadlock situations to force a worker rest= art and to fail+retry the batch).

From the logs, one such IBackingMap worker death (and subseq= uent restart) resulted in the Kafka spout re-emitting the pending tuple:
=A0=A0=A0 2014-03-22 16:26:43 s.k.t.TridentKafkaEmitter [INFO] re-emit= ting batch, attempt 29698959:736

This is of course the normal behavior of a transa= ctional topology, but this is the first time I've encountered a case of= a batch retrying indefinitely. This is especially suspicious since the top= ology has been running fine for 20 days straight, re-emitting batches and r= estarting IBackingMap workers quite a number of times.

I can see in my IBackingMap backing SQL database that the batch with th= e exact txid value 29698959 has been committed -- but I suspect that could = come from another BackingMap, since there are two BackingMap instances runn= ing (paralellismHint 2).

However, I have no idea why the batch is being retried= indefinitely now nor why it hasn't been successfully acked by Trident.=

Any suggestions on the area (topology component) to focus my resear= ch on?

Thanks,

On Wed, = Mar 26, 2014 at 5:32 PM, Danijel Schiavuzzi <danije= l@schiavuzzi.com> wrote:
Hello,

I= 'm having problems with my transactional Trident topology. It has been = running fine for about 20 days, and suddenly is stuck processing a single b= atch, with no tuples being emitted nor tuples being persisted by the Triden= tState (IBackingMap).

It's a simple topology which consumes messages off a Kafka queue. T= he=20 spout is an instance of storm-kafka-0.8-plus=20 TransactionalTridentKafkaSpout and I use the trident-mssql transactional TridentState implementation to persistentAggregate() data into a SQL datab= ase.

In Zookeeper I can see Storm is re-trying a batch, i= .e.

=A0=A0=A0=A0 "/transactional/<myTopologyName>/coordin= ator/currattempts" is "{"29698959":6487}"

... and the attempt count keeps increasing. It se= ems the batch with txid 29698959 is stuck, as the attempt count in Zookeepe= r keeps increasing -- seems like the batch isn't being acked by Trident= and I have no idea why, especially since the topology has been running suc= cessfully the last 20 days.

I did rebalance the topology on one occasion, after which it= continued running normally. Other than that, no other modifications were d= one. Storm is at version 0.9.0.1.

Any hints on= how to debug the stuck topology? Any other useful info I might provide?

Thanks,

-- <= br>Danijel Schiavuzzi

E: = danijel@schiavuzzi.com
W: ww= w.schiavuzzi.com
T: +3859890= 35562
Skype: danijel.schiavuzzi



--
Danijel Sch= iavuzzi

E: danije<= /font>


-= -
Danijel Schiavuzzi

E: danijel@schi= avuzzi.com
W: ww= w.schiavuzzi.com
T: +3859890= 35562
Skype: danijels7




--
Danijel Sch= iavuzzi

E: danijel@schiavuzzi.com W: ww= w.schiavuzzi.com
T: +3859890= 35562
Skype: danijels7



--001a11353c74426d8804f6a31ec6--