hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ming Li <...@pivotal.io>
Subject Re: HAWQ standby master sync process
Date Mon, 05 Sep 2016 15:41:21 GMT
Hi,

The general idea please refer to PostgreSQL:
https://www.pgcon.org/2008/schedule/attachments/61_Synchronous%20Log%20Shipping%20Replication.pdf


Here just share some info about standby code.

The standby related code is here:
src/backend/postmaster/walredoserver.c
src/backend/postmaster/walsendserver.c

Global pic:
- Backend generate WAL and pass it to the forked process "WAL Sender",  the
calling stack is: XLogQDMirrorWrite() => WalSendServerClientSendRequest()

- "WAL sender" process will be forked up and loop for processing request
and response, the calling stack is:
walsendserver_forkexec() -> walsendserver_start() -> ServiceMain() ->
ServiceListenLoop() -> ServiceProcessRequest() ->
serviceConfig->ServiceRequest()
-> WalSendServer_ServiceRequest()

- "WAL Sender" send WAL to "WAL Receiver" which is on the standby node, the
calling stack is:
WalSendServer_ServiceRequest() => WalSendServerDoRequest() =>
disconnectMirrorQD_SendClose() => write_qd_sync() => PQsendQuery()

- On the standby side, all API are similar,  e.g. walredoserver_forkexec()
vs walsendserver_forkexec()

Hope it helps you! ~_~



On Thu, Aug 11, 2016 at 1:09 AM, Kyle Dunn <kdunn@pivotal.io> wrote:

> Hello,
>
> I'm investigating DR options for HAWQ and was curious about the existing
> master catalog synchronization process. My question is mainly around what
> this process does at a high level and where I might look in the code base
> or management tools to see about extending it for additional standby
> masters (e.g. one in a geographically distant data center and/or different
> logical HAWQ cluster). The assumption is the HDFS blocks would be
> replicated by something like distcp via Falcon.
>
> I believe there are obvious things to address like DFS / namenode URI
> parameters, FQDNs, and certainly failure scenarios / edge cases, but I'm
> mainly trying to get a dialog started to see what input, ideas, and
> considerations others have. One thing I'm specifically interested in is
> whether / how WAL can be used (@Keaton).
>
>
> Thanks,
> Kyle
> --
> *Kyle Dunn | Data Engineering | Pivotal*
> Direct: 303.905.3171 <3039053171> | Email: kdunn@pivotal.io
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message