storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enno Shioji <enno.shi...@peerindex.com>
Subject Re: Deactivate topology incase of network issues!!!
Date Mon, 10 Feb 2014 10:17:49 GMT
T.b.h. killing your topology on a single SQL exception is prob. a bad idea,
they can happen time to time and you don't want to kill your system because
of that.

I might look at setting TOPOLOGY_MAX_SPOUT_PENDING to some manageable
number so that you won't have large numbers of tuples flying in your system
and thus slashing your cluster's performance. That way when MySQL comes
back up, the topology can resume processing. IMO that's much preferable.


On Mon, Feb 10, 2014 at 9:53 AM, Chitra Raveendran <
chitra.raveendran@flutura.com> wrote:

> Thanks a lot Bijoy
>
> I will test that out and get back to you.
>
> Regards
> Chitra
>
>
> On Mon, Feb 10, 2014 at 2:14 PM, bijoy deb <bijoy.computers@gmail.com>wrote:
>
>> Hi,
>>
>> You can catch the exception in your bolt (which is connecting to sql
>> server) and in the catch block you can kill or deactivate the topology
>> using Nimbus client,as below:
>>
>> NimbusClient client = .......
>>
>> client.killTopology(
>> "topologyName");//or, client.deactivate("topologyName");
>>
>>
>>
>> Thanks
>>
>> Bijoy
>>
>>
>> On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
>> chitra.raveendran@flutura.com> wrote:
>>
>>> Hi
>>>
>>> I have a Realtime data crunching pipeline which wherein storm consumes
>>> data from kafka, we do some processing on this data (basically counts and
>>> aggregations) and store the results into a MySql database.( I use the
>>> default Kafka spout.)
>>>
>>> Over the weekend some network related issues led to MySQL server crash,
>>> as a result storm kept re-sending the messages, the processing in the storm
>>> supervisors increased drastically and CPU usage hit 100%. This in turn led
>>> to some supervisors falling out of the cluster and trying to restart.
>>>
>>> How would I be able to *handle* such an unforeseen situation? Is there
>>> any way through which storm would understand that there is a MySql server
>>> issue and stop re-sending data?
>>>
>>> Is there anyway that this exception can be caught and in *topology be
>>> deactivated* for a while?
>>>
>>> Regards,
>>> Chitra
>>>
>>>
>>
>
>
>

Mime
View raw message