hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Sharma <ssha...@pivotal.io>
Subject Re: Hawq standby sync fails if there are existing connections to master
Date Tue, 14 Nov 2017 03:06:08 GMT
Thanks Radar, fixed this under PR -
https://github.com/apache/incubator-hawq/pull/1311

On Mon, Nov 13, 2017 at 6:32 PM, Radar Lei <rlei@pivotal.io> wrote:

> Hi Shubham,
>
> The behavior is intentional but use '-M' option to force it should be good.
> Thanks.
>
> Regards,
> Radar
>
> On Tue, Nov 14, 2017 at 10:05 AM, Shubham Sharma <ssharma@pivotal.io>
> wrote:
>
> > To close this thread, the source of this issue appears to be
> > tools/bin/hawq_ctl method _resync_standby. When this is called the
> command
> > formation does not include stop_mode options as passed to the arguments.
> >
> >  def _resync_standby(self):
> >         logger.info("Re-sync standby")
> >         cmd = "%s; hawq stop master -a;" % source_hawq_env
> >         check_return_code(local_ssh(cmd, logger), logger, "Stop hawq
> > cluster failed, exit")
> >         ......
> >         ......
> >
> > JIRA ID - HAWQ-1549
> >
> > On Mon, Nov 13, 2017 at 5:56 PM, Shubham Sharma <ssharma@pivotal.io>
> > wrote:
> >
> > > Thanks, I think I found out where the issue is. Will create a JIRA and
> > > submit PR.
> > >
> > > On Mon, Nov 13, 2017 at 5:54 PM, Lei Chang <chang.lei.cn@gmail.com>
> > wrote:
> > >
> > >> oh, yes, make sense, I overlooked the -M option here.
> > >>
> > >> Here we should respect the input "-M" option.
> > >>
> > >> Cheers
> > >> Lei
> > >>
> > >>
> > >>
> > >> On Tue, Nov 14, 2017 at 8:46 AM, Shubham Sharma <ssharma@pivotal.io>
> > >> wrote:
> > >>
> > >> > Thanks for the response Lei.
> > >> >
> > >> > I understand this completely and agree with the behavior that we
> > should
> > >> not
> > >> > brutally terminate connections.
> > >> >
> > >> > However, the case here is, am deliberately trying to use stop mode
> > using
> > >> > command `hawq init standby -n -v -M fast`. At this point as a user
I
> > >> > understand that connections will be terminated without warning. When
> > >> > passing option `-M fast` hawq should not interrupt me.
> > >> >
> > >> > Basically, hawq command line is not respecting `-M fast`. Whereas
> hawq
> > >> init
> > >> > standby --help documents this option. Let me know if this makes
> sense.
> > >> >
> > >> >
> > >> > hawq init standby --help
> > >> >
> > >> > Usage: HAWQ management scripts options
> > >> >
> > >> >
> > >> > Options:
> > >> >
> > >> >   -h, --help            show this help message and exit
> > >> >
> > >> >   -a, --prompt          Execute automatically
> > >> >
> > >> >   -M STOP_MODE, --mode=STOP_MODE
> > >> >
> > >> >                         HAWQ stop mode: smart/fast/immediate
> > >> >
> > >> >
> > >> > On Mon, Nov 13, 2017 at 4:32 PM, Lei Chang <chang.lei.cn@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi Shubham,
> > >> > >
> > >> > > The behavior is intentional. If there are connections when HAWQ
> init
> > >> > > standby, it is better to warn the client instead of cutting the
> > >> > connections
> > >> > > brutally.
> > >> > >
> > >> > > Cheers
> > >> > > Lei
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Nov 13, 2017 at 11:58 AM, Shubham Sharma <
> > ssharma@pivotal.io>
> > >> > > wrote:
> > >> > >
> > >> > > > Hello folks,
> > >> > > >
> > >> > > > Recently observed a behaviour while re-syncing standby from
hawq
> > >> > command
> > >> > > > line.
> > >> > > >
> > >> > > > Here are the reproduction steps -
> > >> > > >
> > >> > > > 1 - Open a client connection to hawq using psql
> > >> > > > 2 - From a different terminal run command - hawq init standby
-n
> > -v
> > >> -M
> > >> > > fast
> > >> > > > 3 - Standby resync fails with error
> > >> > > >
> > >> > > >
> > >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[
> WARNING]:-There
> > >> are
> > >> > > > other connections to this instance, shutdown mode smart
aborted
> > >> > > >
> > >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[
> WARNING]:-Either
> > >> > > > remove connections, or use 'hawq stop master -M fast' or
'hawq
> > stop
> > >> > > > master -M immediate'
> > >> > > >
> > >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[WARNING]:-See
> > hawq
> > >> > > > stop --help for all options
> > >> > > >
> > >> > > > 20171113:03:49:21:158354 hawq_stop:hdp3:gpadmin-[ERROR]:-Active
> > >> > > > connections. Aborting shutdown...
> > >> > > >
> > >> > > > 20171113:03:49:21:158143 hawq_init:hdp3:gpadmin-[ERROR]:-Stop
> > hawq
> > >> > > > cluster failed, exit
> > >> > > >
> > >> > > > 4 - My understanding is when -M (stop mode) is passed it
should
> > >> > terminate
> > >> > > > existing client connections. Also, it seems like a good
practice
> > to
> > >> > > > terminate client connections before standby master resync.
> > >> > > >
> > >> > > > Is this an expected behavior in hawq ? If not, I can open
a JIRA
> > and
> > >> > work
> > >> > > > on a pull request to fix this.
> > >> > > >
> > >> > > > Looking forward to your thoughts on this.
> > >> > > > ‚Äč
> > >> > > > --
> > >> > > > Regards,
> > >> > > > Shubham Sharma
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Shubham Sharma
> > >> > Staff Customer Engineer
> > >> > Pivotal Global Support Services
> > >> > ssharma@pivotal.io
> > >> > Direct Tel: +1(510)-304-8201
> > >> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > >> > Out of Office Hours Contact +1 877-477-2269
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Shubham Sharma
> > > Staff Customer Engineer
> > > Pivotal Global Support Services
> > > ssharma@pivotal.io
> > > Direct Tel: +1(510)-304-8201 <(510)%20304-8201>
> > > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > > Out of Office Hours Contact +1 877-477-2269 <(877)%20477-2269>
> > >
> >
> >
> >
> > --
> > Regards,
> > Shubham Sharma
> > Staff Customer Engineer
> > Pivotal Global Support Services
> > ssharma@pivotal.io
> > Direct Tel: +1(510)-304-8201
> > Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
> > Out of Office Hours Contact +1 877-477-2269
> >
>



-- 
Regards,
Shubham Sharma
Staff Customer Engineer
Pivotal Global Support Services
ssharma@pivotal.io
Direct Tel: +1(510)-304-8201
Office Hours: Mon-Fri 9:00 am to 5:00 pm PDT
Out of Office Hours Contact +1 877-477-2269

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message