db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Korneliussen <Andreas.Kornelius...@Sun.COM>
Subject Re: conflict detection strategies
Date Mon, 20 Feb 2006 09:20:27 GMT
Mike Matrigali wrote:
> What is your definition of a "valid" RowLocation?  Mine is:
>
> 1) A non-deleted RowLocation is guaranteed to point at the intended
>    record until a delete is executed, by a client of the access method.
>    This part requires no locking, and is the current protocol.
> 
> 2) A row lock is required to prevent a delete of the row by another
>    transaction.  There is no way to prevent a delete of the RowLocation
>    by the same transaction.  This is the current protocol.
> 
> 3) I think SUR requires that upon committed delete access to the
>    RowLocation always return some sort of failure, and never access
>    to a different row.  Truncate currently breaks this, for
>    conglomerates that guarantee non-reusable RowLocations.  In current
>    access methods this could be enforced this while holding a
>    easily if the table level intent
>    lock requirement is added.
>    I would be comfortable adding this to store contract.  It
>    seems reasonable and allows coordination through locking.
>
Yes, I agree with you definition, and I would appreciate if you could 
add the table intent lock requirement to the contract.

> Note this does not adress other current client usages of the access
> methods.  But I believe that all those clients could easily agree
> to the table intent lock protocol.  This would mean that any client
> that wanted to break this protol must get an exclusive table lock
> first.  I believe all those clients already do, but for different
> reasons.
> 
> Does this solve your non-holdable case?  The holdable case is a 
> different beast, and should be a different thread.
> 
Yes, as far as I can see, this solves the non-holdable case.

And to be entirely safe, would it be possible to add a requirement that 
truncate/defragment and other compress operations always should run in 
autocommit mode ?

I am thinking of the cases were a user could run these in the same 
transaction as they have an updatable cursor ? If they are in autocommit 
mode, I think the cursor (non-holdable) will be closed.

Andreas





> 
> 
> Andreas Korneliussen wrote:
> 
>> Mike Matrigali wrote:
>>
>>> It looks like I may have been thinking about future directions vs.
>>> current reality.  The question still is what should the contract
>>> be, rather than what a specific implementation currently provides.
>>>
>>
>> I agree. I think the contract could be one of these:
>>
>> 1. A RowLocation is valid as long as the transaction has a table 
>> intent lock
>> 2. A RowLocation is valid as long as the transaction has a row lock 
>> for the row.
>>
>> Both contracts have different tradeoffs, I guess. for store, alt. 2 
>> gives more flexibility in future online compress operations, however I 
>> think it would require SURs to set and hold locks for all isolation 
>> levels.
>>
>> JavaDoc for RowLocation do say:
>>
>>   See the conglomerate implementation specification for
>>   information about the conditions under which a row location
>>   remains valid.
>>
>> So currently, it is implementation defined.
>>
>>
>> Andreas
>>
>>>
>>>
>>> Andreas Korneliussen wrote:
>>>
>>>> Mike Matrigali wrote:
>>>>
>>>>> I posted some questions about how the delete/update is done, those
>>>>> answers would help me understand better what is needed for a solution.
>>>>>
>>>>> I am going to start a separate thread concentrating on RowLocation
>>>>> guarantees from store.
>>>>>
>>>> That is great.
>>>>
>>>>> Some other answers below.
>>>>>
>>>>> Andreas Korneliussen wrote:
>>>>>
>>>>>
>>>>>> Mike Matrigali wrote:
>>>>>>
>>>>>>
>>>>>>> There are very few cross thread dependencies not managed by locks
>>>>>>> currently.  These things add extra complications to current and
>>>>>>> future code.  Also I want to understand clearly the new restrictions
>>>>>>> be imposted on the access methods (both current and possible
>>>>>>> future).  In the future we would like to do more automatic space
>>>>>>> reclamation as part of the zero-admin goal, the ability to do
this
>>>>>>> in the future internal to access methods is probably affected
by
>>>>>>> the proposals here.
>>>>>>>
>>>>>>
>>>>>> I think the complexities are there already. It seems very hard to

>>>>>> know
>>>>>> under exactly which conditions, a RowLocation remains valid.  We
have
>>>>>> assumed that the RowLocation will remain valid as long as we hold
a
>>>>>> table intent lock (valid means that it either points to the same

>>>>>> row or
>>>>>> to a deleted row).
>>>>>>
>>>>>> That is what we concluded from the previous discusssion about 
>>>>>> validity
>>>>>> of RowLocation. If you in the future need to make code, or there

>>>>>> already
>>>>>> is code, which breaks this assumption, we would need to know which

>>>>>> other
>>>>>> mechanisms we should use to either verify that the RowLocation is

>>>>>> valid,
>>>>>> or to block the system to make it invalid.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> There is already code that breaks this assumption, the in place 
>>>>> compress
>>>>> table.  It currently is only executed as a call to a system procedure:
>>>>> http://db.apache.org/derby/docs/dev/ref/rrefproceduresinplacecompress.html

>>>>>
>>>>>
>>>>> But the hope was in the future to do the same kind of work that this
>>>>> procedure does, internally in background.
>>>>>
>>>>> As you have already determined that off-line compress is even more 
>>>>> of a
>>>>> problem, but it does run under an exclusive table lock.  After it runs
>>>>> the container you were connected to does not even exist any more.
>>>>>
>>>>
>>>> The online compress seems to require table-exclusive locks when 
>>>> passing the defragment or truncate_end argument.
>>>> There are two testcases in jdbcapi/ConcurrencyTest which test this 
>>>> (testDefragmentDuringScan and testTruncateDuringScan).
>>>>
>>>> The tests first deletes all rows except the first and the last. Then 
>>>> it commits. Then it opens a SUR in read-uncommitted mode. Read all 
>>>> records into the resultset, and position the cursor to afterlast.
>>>> Then it runs the defragment or truncate. The test will hang waiting 
>>>> for a lock. Finally, it updates the rows correctly.
>>>>
>>>> Output from the tests (modified the test slightly to get more output):
>>>> T1: delete records
>>>> T1: commit
>>>> T2: Read next Tuple:(0,0,17)
>>>> T2: Read next Tuple:(9,9,35)
>>>> T3: call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE
>>>> T3: DEFRAGMENT
>>>> ERROR 40XL1: A lock could not be obtained within the time requested
>>>> .... 
>>>> org.apache.derby.impl.store.access.RAMTransaction.openScan(RAMTransaction.java:1582)

>>>>
>>>>         at 
>>>> org.apache.derby.iapi.db.OnlineCompress.setup_indexes(OnlineCompress.java:605)

>>>>
>>>>         at 
>>>> org.apache.derby.iapi.db.OnlineCompress.defragmentRows(OnlineCompress.java:359)

>>>>
>>>>         at 
>>>> org.apache.derby.iapi.db.OnlineCompress.compressTable(OnlineCompress.java:227)

>>>>
>>>>         at 
>>>> org.apache.derby.catalog.SystemProcedures.SYSCS_INPLACE_COMPRESS_TABLE(SystemProcedures.java:858)

>>>>
>>>> ...
>>>>        at 
>>>> org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testCompressDuringScan(ConcurrencyTest.java:777)

>>>>
>>>>        at 
>>>> org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testDefragmentDuringScan(ConcurrencyTest.java:712)

>>>>
>>>> ...
>>>>
>>>> T3: got expected exception
>>>> T1: Read first Tuple:(0,0,17)
>>>> T1: updateInt(2, 3);
>>>> T1: updateRow()
>>>> T1: Read last Tuple:(9,9,35)
>>>> T1: updateInt(2, 3);
>>>> T1: updateRow()
>>>> T1: commit
>>>> T4: select * from table
>>>> T4: Read next Tuple:(0,3,17)
>>>> T4: Read next Tuple:(9,3,35)
>>>>
>>>> So to me it seems that our assumptions are correct.. Only a purge is 
>>>> allowed with row-level locking, online compress and online 
>>>> defragment seems to be blocked by an open cursor on the table.
>>>> Maybe that is not how online compress was intended to be ?
>>>>
>>>> Andreas
>>>>
>>>>
>>>>>
>>>>>> Locks can be used to manage cross thread dependencies, however 
>>>>>> they are
>>>>>> bound to the transaction, and therefore does not help very much for
>>>>>> cursors held across commits. So if the only mechanism we can have
to
>>>>>> ensure that the RowLocations are valid, is by the means of locks,
I
>>>>>> doubt we can support the feature of scrollable insensitive *holdable*
>>>>>> updatable resultset.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I agree, I don't believe derby is currently architected to correctly
>>>>> implement "holdable" SUR.  I don't think the outside event driven
>>>>> approach is the right way to go.
>>>>>
>>>>>>> It is true that the current access methods don't reuse row locations
>>>>>>> until a table level lock is granted.  But your project would
be the
>>>>>>> first dependency on this outside of the access method 
>>>>>>> implementations
>>>>>>> themselves.  It is very clear the contract that the access methods
>>>>>>> have with their clients while locks are held on the data they
are
>>>>>>> looking at, what you are proposing is a contract on unlocked
data.
>>>>>>>
>>>>>>
>>>>>> I guess we are the first to use RowLocation without holding a lock
on
>>>>>> the row. This is necessary, unless we simply make SUR cursors set

>>>>>> locks
>>>>>> for all rows in the cursor independent from isolation level.
>>>>>>
>>>>>>
>>>>>>> Note that the current "in-place" compress will MOVE rows from
one
>>>>>>> row location to another if one does not have a row lock on the
row.
>>>>>>> This is done in the 2nd phase and only holds an intent lock,
and
>>>>>>> exclusive row locks on the rows being moved.
>>>>>>> The off-line compress only does work under an X table lock.
>>>>>>> So the row that you are updating actually will exist in the table,
>>>>>>> but currently you will request the old location and will get
back
>>>>>>> a delete row indicator.  I think because of this option 1 does
not
>>>>>>> work.
>>>>>>>
>>>>>>
>>>>>> Are you saing that RowLocations can be invalidated by "in-place"
>>>>>> compress even if we hold a Table intent lock ?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> The "in-place" compress will do the following take a row a row 
>>>>> location
>>>>> N and do the
>>>>> following:
>>>>>     insert row at new row location M
>>>>>     delete row at old row location N
>>>>>     possibly purge old row location
>>>>>
>>>>> I don't understand how this will affect your code, it depends how you
>>>>> search for the row, and what errors are ok to throw.
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
>>
> 


Mime
View raw message