Hi
I have a Realtime data crunching pipeline which wherein storm consumes data
from kafka, we do some processing on this data (basically counts and
aggregations) and store the results into a MySql database.( I use the
default Kafka spout.)
Over the weekend some network related issues led to MySQL server crash, as
a result storm kept re-sending the messages, the processing in the storm
supervisors increased drastically and CPU usage hit 100%. This in turn led
to some supervisors falling out of the cluster and trying to restart.
How would I be able to *handle* such an unforeseen situation? Is there any
way through which storm would understand that there is a MySql server issue
and stop re-sending data?
Is there anyway that this exception can be caught and in *topology be
deactivated* for a while?
Regards,
Chitra
|