Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Date: Fri, 9 Mar 2012 15:34:57 +0000 (UTC)
From: "Rick Hillegas (Commented) (JIRA)" <jira@apache.org>
To: derby-dev@db.apache.org
Message-ID: 
 <342767240.43714.1331307297705.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1424585030.4824.1317678574246.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (DERBY-5443) reduce number of times sequence
 updater does it work on user thread rather than nested user thread.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/DERBY-5443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226142#comment-13226142 ] 

Rick Hillegas commented on DERBY-5443:
--------------------------------------

Thanks for thinking about my proposal, Mike. You are right that this JIRA started out being focussed on a concurrency problem and I am also trying to tackle a related correctness problem. I am concerned that solving the concurrency problem will be throwaway work when we get around to solving the correctness problem.

Concerning the concurrency problem: I think that we will still suffer pile-ups even if we reduce the number of times that work gets escalated to the user transaction. My experience is that once even one escalation occurs, then all other users of the identity/sequence escalate and pile-up behind one another and concurrency degrades to single-threaded behavior.

That's the concurrency problem. But there's also the correctness problem described by DERBY-5493. This happens when pre-allocation occurs in an escalated user transaction. If that transaction rolls back later on, then the on-disk state of the sequence/identity is no longer in sync with the in-memory state being managed by the SequenceGenerator.

My proposal is meant to solve both the concurrency and the correctness problems.

You are right, the proposal is complicated and that is a serious cause for concern. A simpler proposal would be better. I'm all ears.

Concerning the CREATE and DROP TABLE behavior of the proposal. You are right that CREATE TABLE has to do work in IC_TRAN as well as the user tran. This is how I see CREATE TABLE working:

1) All of the existing writes will continue to happen in the user transaction.

2) In addition, a row will be added to INVISIBLE_CONGLOMERATE in the IC_TRAN. The user who created the table should be able to use the SequenceGenerator right away, even if the CREATE TABLE user transaction has not committed. No one else will even try to access the SequenceGenerator  until the user transaction commits and the table becomes visible to them. If the user transaction rolls back and the table disappears, then there will be an orphaned tuple in INVISIBLE_CONGLOMERATE.

I hope that DROP TABLE is even simpler. I don't think that it needs to drop the corresponding tuple in INVISIBLE_CONGLOMERATE. The tuple will just be orphaned. Since it is keyed by the table's UUID, no one will try to access it after the DROP TABLE commits and the table disappears.

Of course, there is a garbage collection problem here. We may want to cleanup the orphaned tuples; maybe at upgrade time; maybe triggered by some other event.

Thanks,
-Rick

                
> reduce number of times sequence updater does it work on user thread rather than nested user thread.
> ---------------------------------------------------------------------------------------------------
>
>                 Key: DERBY-5443
>                 URL: https://issues.apache.org/jira/browse/DERBY-5443
>             Project: Derby
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 10.9.0.0
>            Reporter: Mike Matrigali
>            Priority: Minor
>         Attachments: blockingDDL.sql
>
>
> Currently the Sequence updater tries to do the system catalog update as part of the user thread, but in a nested user transaction.  When this works
> all is well as the nested user transaction is immediately committed and thus the throughput of all threads depending on allocating sequences is
> optimized.  
> In order to be able to commit the nested writable transaction independently the lock manager must treat the parent and nested transactions as two
> independent transactions and locks held by the parent will thus block the child.  And in effect any lock that is blocked by the parent is a deadlock,
> but the lock manager does not understand this relationship and thus only will timeout and not recognize the implicit deadlock.
> Only 2 cases come to mind of the parent blocking the child in this manner for sequences:
> 1) ddl like create done in transaction followed by inserts into the table requiring sequence update.
> 2) users doing jdbc data dictionary lookups in a multistatment transaction resulting in holding locks on the system catalog rows and subsequently
>     doing inserts into the table requiring sequence updates.
> The sequence updater currently never waits for a lock in the nested transaction and assumes any blocked lock is this parent deadlock case.  It
> then falls back on doing the update in tranaction and then the system catalog lock remains until the user transaction commits which could then
> hold hostage all other inserts into the table.  This is ok in the above 2 cases as there is not any other choice since the user transaction is already
> holding the system hostage.  
> The problem is the case where it was not a deadlock but just another thread trying to do the sequence update.  In this case the thread should
> not be getting locks on the user thread.  
> I am not sure best way to address this project but here are some ideas:
> 1) enhance lock manager to recognize the deadlock and then change to code to somehow do an immediately deadlock check for internal 
>     nested transactions, no matter what the system default is.  Then the code should go ahead and use the system wait timeout on this lock
>     and only fall over to using user transaction for deadlock (or maybe even throw a new "self deadlock" error that would only be possible for
>     internal transactions).
> 2) somehow execute the internal system catalog update as part of a whole different transaction in the system.   Would need a separate context.
>     Sort of like the background daemon threads.  Then no self deadlock is possible and it could just go ahead and wait.  The downside is that then
>     the code to "wait" for a new sequence becomes more complicated as it has to wait for an event from another thread.  But seems like it could
>     designed with locks/synchonization blocks somehow.  
> 3) maybe add another lock synchronization that would only involve threads updating the sequences.  So first an updater would request the
>     sequence updater lock (with a key specific to the table and a new type) and it could just wait on it.  It should never be held by parent
>     transaction.  Then it would still need the catalog row lock to do the update.  I think with proper ordering this would insure that blocking on
>     the catalog row lock would only happen in the self deadlock case.  
> Overall this problem is less important as the size of the chunk of sequence is tuned properly for the application, and ultimately best if derby
> autotuned the chunk.  There is a separate jira for auto tuning: DERBY-5295

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira