db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Korneliussen <Andreas.Kornelius...@Sun.COM>
Subject Re: conflict detection strategies
Date Fri, 17 Feb 2006 17:32:01 GMT
Mike Matrigali wrote:
> It looks like I may have been thinking about future directions vs.
> current reality.  The question still is what should the contract
> be, rather than what a specific implementation currently provides.
> 

I agree. I think the contract could be one of these:

1. A RowLocation is valid as long as the transaction has a table intent lock
2. A RowLocation is valid as long as the transaction has a row lock for 
the row.

Both contracts have different tradeoffs, I guess. for store, alt. 2 
gives more flexibility in future online compress operations, however I 
think it would require SURs to set and hold locks for all isolation levels.

JavaDoc for RowLocation do say:

   See the conglomerate implementation specification for
   information about the conditions under which a row location
   remains valid.

So currently, it is implementation defined.


Andreas

> 
> 
> Andreas Korneliussen wrote:
> 
>> Mike Matrigali wrote:
>>
>>> I posted some questions about how the delete/update is done, those
>>> answers would help me understand better what is needed for a solution.
>>>
>>> I am going to start a separate thread concentrating on RowLocation
>>> guarantees from store.
>>>
>> That is great.
>>
>>> Some other answers below.
>>>
>>> Andreas Korneliussen wrote:
>>>
>>>
>>>> Mike Matrigali wrote:
>>>>
>>>>
>>>>> There are very few cross thread dependencies not managed by locks
>>>>> currently.  These things add extra complications to current and
>>>>> future code.  Also I want to understand clearly the new restrictions
>>>>> be imposted on the access methods (both current and possible
>>>>> future).  In the future we would like to do more automatic space
>>>>> reclamation as part of the zero-admin goal, the ability to do this
>>>>> in the future internal to access methods is probably affected by
>>>>> the proposals here.
>>>>>
>>>>
>>>> I think the complexities are there already. It seems very hard to know
>>>> under exactly which conditions, a RowLocation remains valid.  We have
>>>> assumed that the RowLocation will remain valid as long as we hold a
>>>> table intent lock (valid means that it either points to the same row or
>>>> to a deleted row).
>>>>
>>>> That is what we concluded from the previous discusssion about validity
>>>> of RowLocation. If you in the future need to make code, or there 
>>>> already
>>>> is code, which breaks this assumption, we would need to know which 
>>>> other
>>>> mechanisms we should use to either verify that the RowLocation is 
>>>> valid,
>>>> or to block the system to make it invalid.
>>>
>>>
>>>
>>> There is already code that breaks this assumption, the in place compress
>>> table.  It currently is only executed as a call to a system procedure:
>>> http://db.apache.org/derby/docs/dev/ref/rrefproceduresinplacecompress.html 
>>>
>>>
>>> But the hope was in the future to do the same kind of work that this
>>> procedure does, internally in background.
>>>
>>> As you have already determined that off-line compress is even more of a
>>> problem, but it does run under an exclusive table lock.  After it runs
>>> the container you were connected to does not even exist any more.
>>>
>>
>> The online compress seems to require table-exclusive locks when 
>> passing the defragment or truncate_end argument.
>> There are two testcases in jdbcapi/ConcurrencyTest which test this 
>> (testDefragmentDuringScan and testTruncateDuringScan).
>>
>> The tests first deletes all rows except the first and the last. Then 
>> it commits. Then it opens a SUR in read-uncommitted mode. Read all 
>> records into the resultset, and position the cursor to afterlast.
>> Then it runs the defragment or truncate. The test will hang waiting 
>> for a lock. Finally, it updates the rows correctly.
>>
>> Output from the tests (modified the test slightly to get more output):
>> T1: delete records
>> T1: commit
>> T2: Read next Tuple:(0,0,17)
>> T2: Read next Tuple:(9,9,35)
>> T3: call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE
>> T3: DEFRAGMENT
>> ERROR 40XL1: A lock could not be obtained within the time requested
>> .... 
>> org.apache.derby.impl.store.access.RAMTransaction.openScan(RAMTransaction.java:1582)

>>
>>         at 
>> org.apache.derby.iapi.db.OnlineCompress.setup_indexes(OnlineCompress.java:605) 
>>
>>         at 
>> org.apache.derby.iapi.db.OnlineCompress.defragmentRows(OnlineCompress.java:359) 
>>
>>         at 
>> org.apache.derby.iapi.db.OnlineCompress.compressTable(OnlineCompress.java:227) 
>>
>>         at 
>> org.apache.derby.catalog.SystemProcedures.SYSCS_INPLACE_COMPRESS_TABLE(SystemProcedures.java:858)

>>
>> ...
>>        at 
>> org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testCompressDuringScan(ConcurrencyTest.java:777)

>>
>>        at 
>> org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testDefragmentDuringScan(ConcurrencyTest.java:712)

>>
>> ...
>>
>> T3: got expected exception
>> T1: Read first Tuple:(0,0,17)
>> T1: updateInt(2, 3);
>> T1: updateRow()
>> T1: Read last Tuple:(9,9,35)
>> T1: updateInt(2, 3);
>> T1: updateRow()
>> T1: commit
>> T4: select * from table
>> T4: Read next Tuple:(0,3,17)
>> T4: Read next Tuple:(9,3,35)
>>
>> So to me it seems that our assumptions are correct.. Only a purge is 
>> allowed with row-level locking, online compress and online defragment 
>> seems to be blocked by an open cursor on the table.
>> Maybe that is not how online compress was intended to be ?
>>
>> Andreas
>>
>>
>>>
>>>> Locks can be used to manage cross thread dependencies, however they are
>>>> bound to the transaction, and therefore does not help very much for
>>>> cursors held across commits. So if the only mechanism we can have to
>>>> ensure that the RowLocations are valid, is by the means of locks, I
>>>> doubt we can support the feature of scrollable insensitive *holdable*
>>>> updatable resultset.
>>>
>>>
>>>
>>>
>>> I agree, I don't believe derby is currently architected to correctly
>>> implement "holdable" SUR.  I don't think the outside event driven
>>> approach is the right way to go.
>>>
>>>>> It is true that the current access methods don't reuse row locations
>>>>> until a table level lock is granted.  But your project would be the
>>>>> first dependency on this outside of the access method implementations
>>>>> themselves.  It is very clear the contract that the access methods
>>>>> have with their clients while locks are held on the data they are
>>>>> looking at, what you are proposing is a contract on unlocked data.
>>>>>
>>>>
>>>> I guess we are the first to use RowLocation without holding a lock on
>>>> the row. This is necessary, unless we simply make SUR cursors set locks
>>>> for all rows in the cursor independent from isolation level.
>>>>
>>>>
>>>>> Note that the current "in-place" compress will MOVE rows from one
>>>>> row location to another if one does not have a row lock on the row.
>>>>> This is done in the 2nd phase and only holds an intent lock, and
>>>>> exclusive row locks on the rows being moved.
>>>>> The off-line compress only does work under an X table lock.
>>>>> So the row that you are updating actually will exist in the table,
>>>>> but currently you will request the old location and will get back
>>>>> a delete row indicator.  I think because of this option 1 does not
>>>>> work.
>>>>>
>>>>
>>>> Are you saing that RowLocations can be invalidated by "in-place"
>>>> compress even if we hold a Table intent lock ?
>>>
>>>
>>>
>>> The "in-place" compress will do the following take a row a row location
>>> N and do the
>>> following:
>>>     insert row at new row location M
>>>     delete row at old row location N
>>>     possibly purge old row location
>>>
>>> I don't understand how this will affect your code, it depends how you
>>> search for the row, and what errors are ok to throw.
>>>
>>
>>
>>
>>
> 


Mime
View raw message