db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Matrigali <mikem_...@sbcglobal.net>
Subject Re: conflict detection strategies
Date Mon, 20 Feb 2006 19:39:44 GMT
I will update the javadoc for the store part with table level intent 
when I get a chance.

As to the autocommit requirement, I will let you propose/implement
that one.  I don't think it is the right way to go, but would not
veto if everyone agreed.  We currently don't have any such requirement
for other statments, and I think it is a hard one to explain to users
and enforce throughout the code.

Andreas Korneliussen wrote:
> Mike Matrigali wrote:
> 
>> What is your definition of a "valid" RowLocation?  Mine is:
>>
>> 1) A non-deleted RowLocation is guaranteed to point at the intended
>>    record until a delete is executed, by a client of the access method.
>>    This part requires no locking, and is the current protocol.
>>
>> 2) A row lock is required to prevent a delete of the row by another
>>    transaction.  There is no way to prevent a delete of the RowLocation
>>    by the same transaction.  This is the current protocol.
>>
>> 3) I think SUR requires that upon committed delete access to the
>>    RowLocation always return some sort of failure, and never access
>>    to a different row.  Truncate currently breaks this, for
>>    conglomerates that guarantee non-reusable RowLocations.  In current
>>    access methods this could be enforced this while holding a
>>    easily if the table level intent
>>    lock requirement is added.
>>    I would be comfortable adding this to store contract.  It
>>    seems reasonable and allows coordination through locking.
>>
> Yes, I agree with you definition, and I would appreciate if you could 
> add the table intent lock requirement to the contract.
> 
>> Note this does not adress other current client usages of the access
>> methods.  But I believe that all those clients could easily agree
>> to the table intent lock protocol.  This would mean that any client
>> that wanted to break this protol must get an exclusive table lock
>> first.  I believe all those clients already do, but for different
>> reasons.
>>
>> Does this solve your non-holdable case?  The holdable case is a 
>> different beast, and should be a different thread.
>>
> Yes, as far as I can see, this solves the non-holdable case.
> 
> And to be entirely safe, would it be possible to add a requirement that 
> truncate/defragment and other compress operations always should run in 
> autocommit mode ?
> 
> I am thinking of the cases were a user could run these in the same 
> transaction as they have an updatable cursor ? If they are in autocommit 
> mode, I think the cursor (non-holdable) will be closed.
> 
> Andreas
> 
> 
> 
> 
> 
>>
>>
>> Andreas Korneliussen wrote:
>>
>>> Mike Matrigali wrote:
>>>
>>>> It looks like I may have been thinking about future directions vs.
>>>> current reality.  The question still is what should the contract
>>>> be, rather than what a specific implementation currently provides.
>>>>
>>>
>>> I agree. I think the contract could be one of these:
>>>
>>> 1. A RowLocation is valid as long as the transaction has a table 
>>> intent lock
>>> 2. A RowLocation is valid as long as the transaction has a row lock 
>>> for the row.
>>>
>>> Both contracts have different tradeoffs, I guess. for store, alt. 2 
>>> gives more flexibility in future online compress operations, however 
>>> I think it would require SURs to set and hold locks for all isolation 
>>> levels.
>>>
>>> JavaDoc for RowLocation do say:
>>>
>>>   See the conglomerate implementation specification for
>>>   information about the conditions under which a row location
>>>   remains valid.
>>>
>>> So currently, it is implementation defined.
>>>
>>>
>>> Andreas
>>>
>>>>
>>>>
>>>> Andreas Korneliussen wrote:
>>>>
>>>>> Mike Matrigali wrote:
>>>>>
>>>>>> I posted some questions about how the delete/update is done, those
>>>>>> answers would help me understand better what is needed for a 
>>>>>> solution.
>>>>>>
>>>>>> I am going to start a separate thread concentrating on RowLocation
>>>>>> guarantees from store.
>>>>>>
>>>>> That is great.
>>>>>
>>>>>> Some other answers below.
>>>>>>
>>>>>> Andreas Korneliussen wrote:
>>>>>>
>>>>>>
>>>>>>> Mike Matrigali wrote:
>>>>>>>
>>>>>>>
>>>>>>>> There are very few cross thread dependencies not managed
by locks
>>>>>>>> currently.  These things add extra complications to current
and
>>>>>>>> future code.  Also I want to understand clearly the new 
>>>>>>>> restrictions
>>>>>>>> be imposted on the access methods (both current and possible
>>>>>>>> future).  In the future we would like to do more automatic
space
>>>>>>>> reclamation as part of the zero-admin goal, the ability to
do this
>>>>>>>> in the future internal to access methods is probably affected
by
>>>>>>>> the proposals here.
>>>>>>>>
>>>>>>>
>>>>>>> I think the complexities are there already. It seems very hard
to 
>>>>>>> know
>>>>>>> under exactly which conditions, a RowLocation remains valid.
 We 
>>>>>>> have
>>>>>>> assumed that the RowLocation will remain valid as long as we
hold a
>>>>>>> table intent lock (valid means that it either points to the same

>>>>>>> row or
>>>>>>> to a deleted row).
>>>>>>>
>>>>>>> That is what we concluded from the previous discusssion about

>>>>>>> validity
>>>>>>> of RowLocation. If you in the future need to make code, or there

>>>>>>> already
>>>>>>> is code, which breaks this assumption, we would need to know

>>>>>>> which other
>>>>>>> mechanisms we should use to either verify that the RowLocation
is 
>>>>>>> valid,
>>>>>>> or to block the system to make it invalid.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> There is already code that breaks this assumption, the in place 
>>>>>> compress
>>>>>> table.  It currently is only executed as a call to a system 
>>>>>> procedure:
>>>>>> http://db.apache.org/derby/docs/dev/ref/rrefproceduresinplacecompress.html

>>>>>>
>>>>>>
>>>>>> But the hope was in the future to do the same kind of work that this
>>>>>> procedure does, internally in background.
>>>>>>
>>>>>> As you have already determined that off-line compress is even more

>>>>>> of a
>>>>>> problem, but it does run under an exclusive table lock.  After it

>>>>>> runs
>>>>>> the container you were connected to does not even exist any more.
>>>>>>
>>>>>
>>>>> The online compress seems to require table-exclusive locks when 
>>>>> passing the defragment or truncate_end argument.
>>>>> There are two testcases in jdbcapi/ConcurrencyTest which test this 
>>>>> (testDefragmentDuringScan and testTruncateDuringScan).
>>>>>
>>>>> The tests first deletes all rows except the first and the last. 
>>>>> Then it commits. Then it opens a SUR in read-uncommitted mode. Read 
>>>>> all records into the resultset, and position the cursor to afterlast.
>>>>> Then it runs the defragment or truncate. The test will hang waiting 
>>>>> for a lock. Finally, it updates the rows correctly.
>>>>>
>>>>> Output from the tests (modified the test slightly to get more output):
>>>>> T1: delete records
>>>>> T1: commit
>>>>> T2: Read next Tuple:(0,0,17)
>>>>> T2: Read next Tuple:(9,9,35)
>>>>> T3: call SYSCS_UTIL.SYSCS_INPLACE_COMPRESS_TABLE
>>>>> T3: DEFRAGMENT
>>>>> ERROR 40XL1: A lock could not be obtained within the time requested
>>>>> .... 
>>>>> org.apache.derby.impl.store.access.RAMTransaction.openScan(RAMTransaction.java:1582)

>>>>>
>>>>>         at 
>>>>> org.apache.derby.iapi.db.OnlineCompress.setup_indexes(OnlineCompress.java:605)

>>>>>
>>>>>         at 
>>>>> org.apache.derby.iapi.db.OnlineCompress.defragmentRows(OnlineCompress.java:359)

>>>>>
>>>>>         at 
>>>>> org.apache.derby.iapi.db.OnlineCompress.compressTable(OnlineCompress.java:227)

>>>>>
>>>>>         at 
>>>>> org.apache.derby.catalog.SystemProcedures.SYSCS_INPLACE_COMPRESS_TABLE(SystemProcedures.java:858)

>>>>>
>>>>> ...
>>>>>        at 
>>>>> org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testCompressDuringScan(ConcurrencyTest.java:777)

>>>>>
>>>>>        at 
>>>>> org.apache.derbyTesting.functionTests.tests.jdbcapi.ConcurrencyTest.testDefragmentDuringScan(ConcurrencyTest.java:712)

>>>>>
>>>>> ...
>>>>>
>>>>> T3: got expected exception
>>>>> T1: Read first Tuple:(0,0,17)
>>>>> T1: updateInt(2, 3);
>>>>> T1: updateRow()
>>>>> T1: Read last Tuple:(9,9,35)
>>>>> T1: updateInt(2, 3);
>>>>> T1: updateRow()
>>>>> T1: commit
>>>>> T4: select * from table
>>>>> T4: Read next Tuple:(0,3,17)
>>>>> T4: Read next Tuple:(9,3,35)
>>>>>
>>>>> So to me it seems that our assumptions are correct.. Only a purge 
>>>>> is allowed with row-level locking, online compress and online 
>>>>> defragment seems to be blocked by an open cursor on the table.
>>>>> Maybe that is not how online compress was intended to be ?
>>>>>
>>>>> Andreas
>>>>>
>>>>>
>>>>>>
>>>>>>> Locks can be used to manage cross thread dependencies, however

>>>>>>> they are
>>>>>>> bound to the transaction, and therefore does not help very much
for
>>>>>>> cursors held across commits. So if the only mechanism we can
have to
>>>>>>> ensure that the RowLocations are valid, is by the means of locks,
I
>>>>>>> doubt we can support the feature of scrollable insensitive 
>>>>>>> *holdable*
>>>>>>> updatable resultset.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> I agree, I don't believe derby is currently architected to correctly
>>>>>> implement "holdable" SUR.  I don't think the outside event driven
>>>>>> approach is the right way to go.
>>>>>>
>>>>>>>> It is true that the current access methods don't reuse row

>>>>>>>> locations
>>>>>>>> until a table level lock is granted.  But your project would
be the
>>>>>>>> first dependency on this outside of the access method 
>>>>>>>> implementations
>>>>>>>> themselves.  It is very clear the contract that the access
methods
>>>>>>>> have with their clients while locks are held on the data
they are
>>>>>>>> looking at, what you are proposing is a contract on unlocked
data.
>>>>>>>>
>>>>>>>
>>>>>>> I guess we are the first to use RowLocation without holding a

>>>>>>> lock on
>>>>>>> the row. This is necessary, unless we simply make SUR cursors
set 
>>>>>>> locks
>>>>>>> for all rows in the cursor independent from isolation level.
>>>>>>>
>>>>>>>
>>>>>>>> Note that the current "in-place" compress will MOVE rows
from one
>>>>>>>> row location to another if one does not have a row lock on
the row.
>>>>>>>> This is done in the 2nd phase and only holds an intent lock,
and
>>>>>>>> exclusive row locks on the rows being moved.
>>>>>>>> The off-line compress only does work under an X table lock.
>>>>>>>> So the row that you are updating actually will exist in the
table,
>>>>>>>> but currently you will request the old location and will
get back
>>>>>>>> a delete row indicator.  I think because of this option 1
does not
>>>>>>>> work.
>>>>>>>>
>>>>>>>
>>>>>>> Are you saing that RowLocations can be invalidated by "in-place"
>>>>>>> compress even if we hold a Table intent lock ?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> The "in-place" compress will do the following take a row a row 
>>>>>> location
>>>>>> N and do the
>>>>>> following:
>>>>>>     insert row at new row location M
>>>>>>     delete row at old row location N
>>>>>>     possibly purge old row location
>>>>>>
>>>>>> I don't understand how this will affect your code, it depends how
you
>>>>>> search for the row, and what errors are ok to throw.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>
> 
> 
> 


Mime
View raw message