storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enno Shioji <enno.shi...@peerindex.com>
Subject Re: Deactivate topology incase of network issues!!!
Date Mon, 10 Feb 2014 11:02:40 GMT
> What would be an appropriate value for TOPOLOGY_MAX_SPOUT_PENDING so that
it would not slow down or affect the present performance (any rough figure)
? There is no default value for this parameter right?

That's a hard question as it depends on your cluster's resource and your
processing... Having said that though, if I had no idea on both parameters
I'd go for something like 5K or 10K and go from there (unless you are doing
something special like producing large number of child tuples off a parent
tuple etc).

Others on this list might be able to give you a better suggestion.






On Mon, Feb 10, 2014 at 10:34 AM, Chitra Raveendran <
chitra.raveendran@flutura.com> wrote:

> Hi
>
> Yeah thanks for that suggestion, killing the process was not exactly what
> I had in mind, But just wanted to test that option too, as the whole
> cluster was affected this weekend.
>
> What would be an appropriate value for TOPOLOGY_MAX_SPOUT_PENDING so that
> it would not slow down or affect the present performance (any rough figure)
> ? There is no default value for this parameter right?
>
>
>
>
>
> On Mon, Feb 10, 2014 at 3:47 PM, Enno Shioji <enno.shioji@peerindex.com>wrote:
>
>> T.b.h. killing your topology on a single SQL exception is prob. a bad
>> idea, they can happen time to time and you don't want to kill your system
>> because of that.
>>
>> I might look at setting TOPOLOGY_MAX_SPOUT_PENDING to some manageable
>> number so that you won't have large numbers of tuples flying in your system
>> and thus slashing your cluster's performance. That way when MySQL comes
>> back up, the topology can resume processing. IMO that's much preferable.
>>
>>
>> On Mon, Feb 10, 2014 at 9:53 AM, Chitra Raveendran <
>> chitra.raveendran@flutura.com> wrote:
>>
>>> Thanks a lot Bijoy
>>>
>>> I will test that out and get back to you.
>>>
>>> Regards
>>> Chitra
>>>
>>>
>>> On Mon, Feb 10, 2014 at 2:14 PM, bijoy deb <bijoy.computers@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> You can catch the exception in your bolt (which is connecting to sql
>>>> server) and in the catch block you can kill or deactivate the topology
>>>> using Nimbus client,as below:
>>>>
>>>> NimbusClient client = .......
>>>>
>>>> client.killTopology(
>>>> "topologyName");//or, client.deactivate("topologyName");
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Bijoy
>>>>
>>>>
>>>> On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
>>>> chitra.raveendran@flutura.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I have a Realtime data crunching pipeline which wherein storm consumes
>>>>> data from kafka, we do some processing on this data (basically counts
and
>>>>> aggregations) and store the results into a MySql database.( I use the
>>>>> default Kafka spout.)
>>>>>
>>>>> Over the weekend some network related issues led to MySQL server
>>>>> crash, as a result storm kept re-sending the messages, the processing
in
>>>>> the storm supervisors increased drastically and CPU usage hit 100%. This
in
>>>>> turn led to some supervisors falling out of the cluster and trying to
>>>>> restart.
>>>>>
>>>>> How would I be able to *handle* such an unforeseen situation? Is
>>>>> there any way through which storm would understand that there is a MySql
>>>>> server issue and stop re-sending data?
>>>>>
>>>>> Is there anyway that this exception can be caught and in *topology be
>>>>> deactivated* for a while?
>>>>>
>>>>> Regards,
>>>>> Chitra
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
>
>
> --
>
> Regards,
>
> *Chitra Raveendran*
> *Data Scientist*
> Mobile: +91 819753660│*Email:* chitra.raveendran@flutura.com
> *Flutura Business Solutions Private Limited – “A Decision Sciences &
> Analytics Company”*│ #693, 2nd Floor, Geetanjali, 15th Cross, J.P Nagar 2
> nd Phase, Bangalore – 560078│
>
>
>

Mime
View raw message