qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Oleksandr Rudyy <oru...@gmail.com>
Subject Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to kill a dead broker
Date Wed, 20 Sep 2017 11:41:32 GMT
Hi Oliver,

By graceful shutdown I meant that Qpid broker jvm exits after successful
closing of all internal objects, finishing/interrupting of all in-progress
tasks and releasing all system resources.
As I mentioned in my previous email, the stop script tries to shutdown
broker gracefully by sending SIGTERM signal. However, the script only waits
for upto 2 seconds for broker to finish graceful shutdown. After that it
sends SIGKILL signal to terminate JVM immediately by OS. The latter can
terminate the JVM abruptly without waiting for broker shutdown to complete
gracefully.

To work around the issue you can use directly "kill -15 <pid>" in your
tests instead of qpid.stop and wait until process finishes. Alternatively,
you can increase SLEEP_DELAY in the stop script to the value sufficient for
the broker to shutdown gracefully.


Kind Regards,
Alex



On 19 September 2017 at 23:04, VERMEULEN Olivier <
Olivier.VERMEULEN@murex.com> wrote:

> Hello,
>
> Thanks for the information.
> There's one thing that's still not clear though, what do you really mean
> by graceful shutdown?
> When you first mentioned this what came to my mind is that the broker will
> finish the tasks at hand before actually shutting down.
> But further testing shows that it's not the case. I did a simple test
> using an infinite loop to send messages to a queue in my broker (with
> auto-ack) and then shutting down the broker with the ReST endpoint (didn't
> try the qpid.stop since I was on Windows but I guess it will be the same).
> As expected the test ends with an exception raised by the producer.send but
> even though we get an exception the last message sent is sometimes
> persisted in the database...
>
> Olivier
>
> -----Original Message-----
> From: Adel Boutros [mailto:adelboutros@live.com]
> Sent: mardi 19 septembre 2017 14:39
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Sorry Alex :(
> ________________________________
> From: Robbie Gemmell <robbie.gemmell@gmail.com>
> Sent: Tuesday, September 19, 2017 1:48:48 PM
> To: users@qpid.apache.org
> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying to
> kill a dead broker
>
> Alex or Oleksandr, rather than Rudyy :)
>
> On 19 September 2017 at 11:22, Adel Boutros <Adelboutros@live.com> wrote:
> > Hello Rudyy,
> >
> >
> > +1 for the enhancements 😊
> >
> >
> > Regards,
> >
> > Adel
> >
> > ________________________________
> > From: Oleksandr Rudyy <orudyy@gmail.com>
> > Sent: Tuesday, September 19, 2017 11:17:15 AM
> > To: users@qpid.apache.org
> > Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps trying
> > to kill a dead broker
> >
> > Hi Olivier,
> >
> > Rob have already provided comprehensive answers on your questions.
> >
> > One thing I would like to add is that both REST API and stop script
> > should shutdown broker gracefully. The script uses signals: it sends
> > SIGTERM to initiate the graceful shutdown. Only if process is still
> > alive after 2 seconds after sending SIGTERM, the SIGKILL signal is
> > sent. I raised
> > QPID-7910 [1] to improve stop script. The possible improvements are
> > listed in the JIRA description.
> >
> > Kind Regards,
> > Alex
> >
> >
> > [1] https://issues.apache.org/jira/browse/QPID-7910
> >
> >
> > On 18 September 2017 at 21:53, VERMEULEN Olivier <
> > Olivier.VERMEULEN@murex.com> wrote:
> >
> >> Hello,
> >>
> >> Following up on Adel's email.
> >> I took a look at the initiateShutdown endpoint you mentioned.
> >> I tested it and it seems to work but I don't see it in any
> >> documentation, not even in the broker apidocs.
> >> Is there a reason for that? Is this feature officially supported?
> >> And one more question, why is the broker's qpid.stop script not using
> >> the same graceful mechanism?
> >>
> >> Thanks,
> >> Olivier
> >>
> >> -----Original Message-----
> >> From: Adel Boutros [mailto:Adelboutros@live.com]
> >> Sent: jeudi 14 septembre 2017 18:34
> >> To: users@qpid.apache.org
> >> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps
> >> trying to kill a dead broker
> >>
> >> Thank you Rudyy,
> >>
> >>
> >> Unfortunately, I don't have any extra  information to share. May be
> >> the stop script should be updated to check output of kill command to
> >> confirm the process is still here.
> >>
> >>
> >> Regards,
> >>
> >> Adel
> >>
> >> ________________________________
> >> From: Oleksandr Rudyy <orudyy@gmail.com>
> >> Sent: Friday, September 8, 2017 1:46:54 PM
> >> To: users@qpid.apache.org
> >> Subject: Re: [Qpid Java Broker 6.0.4] [Linux] Stop script keeps
> >> trying to kill a dead broker
> >>
> >> Hi Adel,
> >> Thanks for reporting the issue. The qpid.stop script might need more
> >> love, though, after sending SIGTERM or SIGKILL event to the broker
> >> process, it waits for 1 second and than verifies that process with
> >> given PID is still reported by ps (ps -e | grep $1 | wc -l). If
> >> process is not reported, no further attempts to send termination
> >> signal is made. It seems that in your case the Broker process was
> >> present in process table. I could be that it became defunctional. You
> >> mentioned that it happens randomly. Do you know what what happens
> >> with the broker and broker jvm? Is broker shutdown gracefully? If
> >> not, it could be an indication of issue with broker shutdown or jvm
> exit.
> >>
> >> Additionally, I would like to point out that you can call Broker REST
> >> API to shutdown the broker (/api/latest/broker/initiateShutdown). As
> >> operation name suggests, it does not shutdown broker immediately but
> >> rather starts the broker  shutdown process and exits. If broker
> >> restart is required, a restart operation can be invoked via REST API
> >> as well (/api/latest/broker/restart).
> >>
> >> Kind Regards,
> >> Alex
> >>
> >>
> >> On 6 September 2017 at 17:21, Adel Boutros <Adelboutros@live.com>
> wrote:
> >>
> >> > Hello,
> >> >
> >> >
> >> > In one of our tests, we were having a random failure. It seems we
> >> > cannot stop a broker correctly.
> >> >
> >> > We have a started broker and we call "bin/qpid.stop $BROKER_PID" to
> >> > stop it. It seems to work from the first time but maybe not fast
> >> > enough because the script keeps trying to kill the broker which is
> >> actually dead.
> >> >
> >> >
> >> > Is this a know issue? Is it fixed on a newer version?
> >> >
> >> >
> >> > Command output
> >> >
> >> > ===============
> >> >
> >> > Waiting 1 second for 514 to exit
> >> > broker/bin/qpid.stop: line 49: kill: (514) - No such process
> >> > Waiting 1 second for 514 to exit
> >> > broker/bin/qpid.stop: line 41: kill: (514) - No such process
> >> > Waiting 1 second for 514 to exit
> >> > broker/bin/qpid.stop: line 41: kill: (514) - No such process
> >> > Waiting 1 second for 514 to exit Stopped trying to kill process:
> >> > 514 Attempted to stop 2 times
> >> >
> >> >
> >> *******************************
> >>
> >> This e-mail contains information for the intended recipient only. It
> >> may contain proprietary material or confidential information. If you
> >> are not the intended recipient you are not authorised to distribute,
> >> copy or use this e-mail or any attachment to it. Murex cannot
> >> guarantee that it is virus free and accepts no responsibility for any
> >> loss or damage arising from its use. If you have received this e-mail
> >> in error please notify immediately the sender and delete the original
> >> email received, any attachments and all copies from your system.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For
> >> additional commands, e-mail: users-help@qpid.apache.org
> >>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org For additional
> commands, e-mail: users-help@qpid.apache.org
>
> *******************************
>
> This e-mail contains information for the intended recipient only. It may
> contain proprietary material or confidential information. If you are not
> the intended recipient you are not authorised to distribute, copy or use
> this e-mail or any attachment to it. Murex cannot guarantee that it is
> virus free and accepts no responsibility for any loss or damage arising
> from its use. If you have received this e-mail in error please notify
> immediately the sender and delete the original email received, any
> attachments and all copies from your system.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message