db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: conflict detection strategies
Date Fri, 17 Feb 2006 18:16:23 GMT
What is your definition of a "valid" RowLocation?  Mine is:

1) A non-deleted RowLocation is guaranteed to point at the intended
    record until a delete is executed, by a client of the access method.
    This part requires no locking, and is the current protocol.

2) A row lock is required to prevent a delete of the row by another
    transaction.  There is no way to prevent a delete of the RowLocation
    by the same transaction.  This is the current protocol.

3) I think SUR requires that upon committed delete access to the
    RowLocation always return some sort of failure, and never access
    to a different row.  Truncate currently breaks this, for
    conglomerates that guarantee non-reusable RowLocations.  In current
    access methods this could be enforced this while holding a
    easily if the table level intent
    lock requirement is added.
    I would be comfortable adding this to store contract.  It
    seems reasonable and allows coordination through locking.

Note this does not adress other current client usages of the access
methods.  But I believe that all those clients could easily agree
to the table intent lock protocol.  This would mean that any client
that wanted to break this protol must get an exclusive table lock
first.  I believe all those clients already do, but for different
reasons.

Does this solve your non-holdable case?  The holdable case is a 
different beast, and should be a different thread.



Andreas Korneliussen wrote:
> Mike Matrigali wrote:
> 
>> It looks like I may have been thinking about future directions vs.
>> current reality.  The question still is what should the contract
>> be, rather than what a specific implementation currently provides.
>>
> 
> I agree. I think the contract could be one of these:
> 
> 1. A RowLocation is valid as long as the transaction has a table intent 
> lock
> 2. A RowLocation is valid as long as the transaction has a row lock for 
> the row.
> 
> Both contracts have different tradeoffs, I guess. for store, alt. 2 
> gives more flexibility in future online compress operations, however I 
> think it would require SURs to set and hold locks for all isolation levels.
> 
> JavaDoc for RowLocation do say:
> 
>   See the conglomerate implementation specification for
>   information about the conditions under which a row location
>   remains valid.
> 
> So currently, it is implementation defined.
> 
> 
> Andreas
> 
>>
>>
>> Andreas Korneliussen wrote:
>>
>>> Mike Matrigali wrote:
>>>
>>>> I posted some questions about how the delete/update is done, those
>>>> answers would help me understand better what is needed for a solution.
>>>>
>>>> I am going to start a separate thread concentrating on RowLocation
>>>> guarantees from store.
>>>>
>>> That is great.
>>>
>>>> Some other answers below.
>>>>
>>>> Andreas Korneliussen wrote:
>>>>
>>>>
>>>>> Mike Matrigali wrote:
>>>>>
>>>>>
>>>>>> There are very few cross thread dependencies not managed by locks
>>>>>> currently.  These things add extra complications to current and
>>>>>> future code.  Also I want to understand clearly the new restrictions
>>>>>> be imposted on the access methods (both current and possible
>>>>>> future).  In the future we would like to do more automatic space
>>>>>> reclamation as part of the zero-admin goal, the ability to do this
>>>>>> in the future internal to access methods is probably affected by
>>>>>> the proposals here.
>>>>>>
>>>>>
>>>>> I think the complexities are there already. It seems very hard to know
>>>>> under exactly which conditions, a RowLocation remains valid.  We have
>>>>> assumed that the RowLocation will remain valid as long as we hold a
>>>>> table intent lock (valid means that it either points to the same 
>>>>> row or
>>>>> to a deleted row).
>>>>>
>>>>> That is what we concluded from the previous discusssion about validity
>>>>> of RowLocation. If you in the future need to make code, or there 
>>>>> already
>>>>> is code, which breaks this assumption, we would need to know which 
>>>>> other
>>>>> mechanisms we should use to either verify that the RowLocation is 
>>>>> valid,
>>>>> or to block the system to make it invalid.
>>>>
>>>>
>>>>
>>>>
>>>> There is already code that breaks this assumption, the in place 
>>>> compress
>>>> table.  It currently is only executed as a call to a system procedure:
>>>> http://db.apache.org/derby/docs/dev/ref/rrefproceduresinplacecompress.html

>>>>
>>>>
>>>> But the hope was in the future to do the same kind of work that this
>>>> procedure does, internally in background.
>>>>
>>>> As you have already determined that off-line compress is even more of a
>>>> problem, but it does run under an exclusive table lock.  After it runs
>>>> the container you were connected to does not even exist any more.
>>>>
>>>
>>> The online compress seems to require table-exclusive locks when 
>>> passing the defragment or truncate_end argument.
>>> There are two testcases in jdbcapi/ConcurrencyTest which test this 
>>> (testDefragmentDuringScan and testTruncateDuringScan).
>>>
>>> The tests first deletes all rows except the first and the last. Then 
>>> it commits. Then it opens a SUR in read-uncommitted mode. Read all 
>>> records into the resultset, and position the cursor to afterlast.
>>> Then it runs the defragment or truncate. The test will hang waiting 
>>> for a lock. Finally, it updates the rows correctly.
>>>
>>> Output from the tests (modified the test slightly to get more output):
>>> T1: delete records
>>> T1: commit
>>> T2: Read next Tuple:(0,0,17)
>>> T2: Read next Tuple:(9,9,35)
>>> T3: call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE
>>> T3: DEFRAGMENT
>>> ERROR 40XL1: A lock could not be obtained within the time requested
>>> .... 
>>> org.apache.derby.impl.store.access.RAMTransaction.openScan(RAMTransaction.java:1582)

>>>
>>>         at 
>>> org.apache.derby.iapi.db.OnlineCompress.setup_indexes(OnlineCompress.java:605)

>>>
>>>         at 
>>> org.apache.derby.iapi.db.OnlineCompress.defragmentRows(OnlineCompress.java:359)

>>>
>>>         at 
>>> org.apache.derby.iapi.db.OnlineCompress.compressTable(OnlineCompress.java:227)

>>>
>>>         at 
>>> org.apache.derby.catalog.SystemProcedures.SYSCS_INPLACE_COMPRESS_TABLE(SystemProcedures.java:858)

>>>
>>> ...
>>>        at 
>>> org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testCompressDuringScan(ConcurrencyTest.java:777)

>>>
>>>        at 
>>> org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testDefragmentDuringScan(ConcurrencyTest.java:712)

>>>
>>> ...
>>>
>>> T3: got expected exception
>>> T1: Read first Tuple:(0,0,17)
>>> T1: updateInt(2, 3);
>>> T1: updateRow()
>>> T1: Read last Tuple:(9,9,35)
>>> T1: updateInt(2, 3);
>>> T1: updateRow()
>>> T1: commit
>>> T4: select * from table
>>> T4: Read next Tuple:(0,3,17)
>>> T4: Read next Tuple:(9,3,35)
>>>
>>> So to me it seems that our assumptions are correct.. Only a purge is 
>>> allowed with row-level locking, online compress and online defragment 
>>> seems to be blocked by an open cursor on the table.
>>> Maybe that is not how online compress was intended to be ?
>>>
>>> Andreas
>>>
>>>
>>>>
>>>>> Locks can be used to manage cross thread dependencies, however they 
>>>>> are
>>>>> bound to the transaction, and therefore does not help very much for
>>>>> cursors held across commits. So if the only mechanism we can have to
>>>>> ensure that the RowLocations are valid, is by the means of locks, I
>>>>> doubt we can support the feature of scrollable insensitive *holdable*
>>>>> updatable resultset.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> I agree, I don't believe derby is currently architected to correctly
>>>> implement "holdable" SUR.  I don't think the outside event driven
>>>> approach is the right way to go.
>>>>
>>>>>> It is true that the current access methods don't reuse row locations
>>>>>> until a table level lock is granted.  But your project would be the
>>>>>> first dependency on this outside of the access method implementations
>>>>>> themselves.  It is very clear the contract that the access methods
>>>>>> have with their clients while locks are held on the data they are
>>>>>> looking at, what you are proposing is a contract on unlocked data.
>>>>>>
>>>>>
>>>>> I guess we are the first to use RowLocation without holding a lock on
>>>>> the row. This is necessary, unless we simply make SUR cursors set 
>>>>> locks
>>>>> for all rows in the cursor independent from isolation level.
>>>>>
>>>>>
>>>>>> Note that the current "in-place" compress will MOVE rows from one
>>>>>> row location to another if one does not have a row lock on the row.
>>>>>> This is done in the 2nd phase and only holds an intent lock, and
>>>>>> exclusive row locks on the rows being moved.
>>>>>> The off-line compress only does work under an X table lock.
>>>>>> So the row that you are updating actually will exist in the table,
>>>>>> but currently you will request the old location and will get back
>>>>>> a delete row indicator.  I think because of this option 1 does not
>>>>>> work.
>>>>>>
>>>>>
>>>>> Are you saing that RowLocations can be invalidated by "in-place"
>>>>> compress even if we hold a Table intent lock ?
>>>>
>>>>
>>>>
>>>>
>>>> The "in-place" compress will do the following take a row a row location
>>>> N and do the
>>>> following:
>>>>     insert row at new row location M
>>>>     delete row at old row location N
>>>>     possibly purge old row location
>>>>
>>>> I don't understand how this will affect your code, it depends how you
>>>> search for the row, and what errors are ok to throw.
>>>>
>>>
>>>
>>>
>>>
>>
> 
> 
> 


Mime
View raw message