zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh R <rake...@huawei.com>
Subject RE: Thread handling
Date Tue, 01 Apr 2014 09:30:38 GMT
Thanks Michi for the reply. 

>>> Does 'shutdown' mean calling System.exit or throwing some kind of exception?
Some applications use ZooKeeper embedded in their JVM, and they might not like ZooKeeper calling
System.exit.

Thats good point. In my mind I'm thinking of graceful shutdown of server by calling ZooKeeperServer#shutdown()
api.

Thanks,
Rakesh

-----Original Message-----
From: mutsuzaki@gmail.com [mailto:mutsuzaki@gmail.com] On Behalf Of Michi Mutsuzaki
Sent: 01 April 2014 14:52
To: dev@zookeeper.apache.org
Cc: user@zookeeper.apache.org
Subject: Re: Thread handling

+1 for shutting down on a critical thread death.

Does 'shutdown' mean calling System.exit or throwing some kind of exception? Some applications
use ZooKeeper embedded in their JVM, and they might not like ZooKeeper calling System.exit.

--Michi

On Mon, Mar 31, 2014 at 9:03 PM, Rakesh R <rakeshr@huawei.com> wrote:
>>>> This is how I handle the critical threads in my client apps that use Zookeeper.
>>>> Keep a reference to the thread and periodically make sure it's still alive
and well - respawn it if it is not.
>
> Thanks Greg for the inputs. Please see ZK-1907, I've included an initial proposal patch
to kick start the discussions.
> Another approach is simply shutdown if a critical thread dies, so the monitoring tool
can easily detect and take necessary actions. The proposed patch is based on this approach.
>
> -Rakesh
>
> -----Original Message-----
> From: Asta, Greg [mailto:greg.asta@omnigon.com]
> Sent: 31 March 2014 23:24
> To: user@zookeeper.apache.org; dev@zookeeper.apache.org
> Subject: RE: Thread handling
>
> " If we have a 'DeathWatcher 'or some other mechanism in place to monitor all the critical
threads. It can take a decision like - bring down the process if required, or shutdown the
quorumpeer and go for LE again etc.
> Now the monitoring or management tool will knows about the situation and can act upon.
>
> Appreciate any thoughts ?"
>
> This is how I handle the critical threads in my client apps that use Zookeeper.  Keep
a reference to the thread and periodically make sure it's still alive and well - respawn it
if it is not.
>
> Thanks,
> Greg
>
>
> -----Original Message-----
> From: Rakesh R [mailto:rakeshr@huawei.com]
> Sent: Thursday, March 27, 2014 10:39 AM
> To: dev@zookeeper.apache.org; user@zookeeper.apache.org
> Subject: Thread handling
>
> Hi All,
>
> Server has many critical threads running and co-ordinating each other like  RequestProcessor
chains et. When going through each threads, most of them having the similar structure like:
>
> public void run() {
>         try {
>               while(running)
>                    // processing logic
>               }
>         } catch (InterruptedException e) {
>             LOG.error("Unexpected interruption", e);
>         } catch (RequestProcessorException e) {
>             LOG.error("Unexpected exception", e);
>         } catch (Exception e) {
>             LOG.error("Unexpected exception", e);
>         }
>         LOG.info("...exited loop!");
> }
>
> I feel, we could improve our threads  in our system. From the design I could see, there
could be a chance of silently leaving the thread in case of any exception(abnormal or any
functional issue too) If this happens in the production, the server would get hanged forever
and will not be able to deliver its role.
>
> If we have a 'DeathWatcher 'or some other mechanism in place to monitor all the critical
threads. It can take a decision like - bring down the process if required, or shutdown the
quorumpeer and go for LE again etc.
> Now the monitoring or management tool will knows about the situation and can act upon.
>
> Appreciate any thoughts ?
>
> Thanks in advance,
> Rakesh R

Mime
View raw message