From derby-dev-return-46614-apmail-db-derby-dev-archive=db.apache.org@db.apache.org Fri Jul 13 10:36:28 2007 Return-Path: Delivered-To: apmail-db-derby-dev-archive@www.apache.org Received: (qmail 10255 invoked from network); 13 Jul 2007 10:36:26 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 13 Jul 2007 10:36:26 -0000 Received: (qmail 6983 invoked by uid 500); 13 Jul 2007 10:36:28 -0000 Delivered-To: apmail-db-derby-dev-archive@db.apache.org Received: (qmail 6770 invoked by uid 500); 13 Jul 2007 10:36:28 -0000 Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: Delivered-To: mailing list derby-dev@db.apache.org Received: (qmail 6761 invoked by uid 99); 13 Jul 2007 10:36:28 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jul 2007 03:36:28 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jul 2007 03:36:24 -0700 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id AE24171420E for ; Fri, 13 Jul 2007 03:36:04 -0700 (PDT) Message-ID: <1491701.1184322964709.JavaMail.jira@brutus> Date: Fri, 13 Jul 2007 03:36:04 -0700 (PDT) From: =?utf-8?Q?J=C3=B8rgen_L=C3=B8land_=28JIRA=29?= To: derby-dev@db.apache.org Subject: [jira] Commented: (DERBY-2926) Replication: Add a log buffer for log records that should be shipped to the slave In-Reply-To: <27751116.1184228044443.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/DERBY-2926?page=3Dcom.atlassian= .jira.plugin.system.issuetabpanels:comment-tabpanel#action_12512423 ]=20 J=C3=B8rgen L=C3=B8land commented on DERBY-2926: -------------------------------------- Hi Mike There is a better description of what we intend to create in DERBY-2872. Ji= ra does not allow multiple layers of subtasks, so I had to use the "is part= of" link. Basically, the goal is to provide asynchronous replication, i.e. replicatio= n where log shipment to the slave is completely decoupled from transactions= on the master. This is even looser synchronization than your third alterna= tive, which I believe is also known as 1-safe replication. The asynchronous replication strategy may result in some lost transactions = when the slave performs fail-over. The amount of lost transactions is, of c= ourse, closely related to how often log shipment is performed. As you menti= on, there is a trade off between how tight the master/slave synchronization= is and the incurred performance degradation. Since log shipping in asynchr= onous replication is completely decoupled from the transactions, this strat= egy should have less performance impact than the alternatives. Although the plan is to add asynchronous replication now, replication with = tighter synchronization should be kept in mind. If possible within reasonab= le increased work, the architecture should easily extend to 1-safe or 2-saf= e (your second alternative) replication later. When it comes to the core replication functionality, there are two things t= he replication master must know about: 1) log writes and 2) log flush. The = log records must (sooner or later) be sent to the slave, hence 1). What to = do with flush calls is up to the replication strategy. For the planned asyn= chronous replication, flush calls can be ignored. For 1-safe and 2-safe rep= lication, flush calls require log shipment as you describe in alternatives = 2 and 3. The current plan for using the log buffer is to append log records to it so= mewhere in LogToFile.appendLogRecord. This is the same method used to appen= d log records to logOut (output stream to the log file; class type LogAcces= sFile). LogAccessFile is implemented with a number byte[] buffers (LogAcces= sFileBuffer), which are ordered in a linked list. LogToFile is the only ent= ry point for log writes, and is therefore easily modifiable for our purpose= .=20 At the other end of the log buffer, a log shipping service will consume log= records. The service should, as you suggest, run as a separate thread. I t= hink DaemonFactory could be useful to create this thread, but that is just = a guess.=20 In the current code, the flush-methods in LogToFile are the only entry poin= ts for transactions to force a log flush (e.g. at commit). Hence, adding fo= rced log shipment to achieve 1 or 2 safe replication later can be easily pu= t in these methods. In the planned asynchronous strategy, log shipment may,= e.g., be based on a timeout; flush calls can be ignored altogether. I hope this clarifies most of your concerns. Does this architecture fit int= o your idea of "tie into the existing log writing code" since log records a= re added to the buffer from the log factory? The single entry point for bot= h log writes and flushes makes this a good place for modifications (for bot= h asynchronous and the x-safe strategies) in my opinion, but there may be g= ood reasons for doing otherwise. > Replication: Add a log buffer for log records that should be shipped to t= he slave > -------------------------------------------------------------------------= -------- > > Key: DERBY-2926 > URL: https://issues.apache.org/jira/browse/DERBY-2926 > Project: Derby > Issue Type: Sub-task > Components: Store > Affects Versions: 10.4.0.0 > Reporter: J=C3=B8rgen L=C3=B8land > Assignee: J=C3=B8rgen L=C3=B8land > > When a Derby instance has the master role for a database, log records are= shipped to the slave to keep it up to date. A buffer is needed because the= log records should not be shipped one at a time. Also, writing the log rec= ords to a buffer instead of sending them immediately removes the network co= mmunication from the critical path for the transaction. --=20 This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.