db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Matrigali (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DERBY-4055) Space may not be reclaimed if row locks are not available after three retries
Date Sat, 14 Feb 2009 00:30:59 GMT

     [ https://issues.apache.org/jira/browse/DERBY-4055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mike Matrigali updated DERBY-4055:
----------------------------------


Wait on the lock or bump on the retry are the only "easy" bug fixes I can 
think of.  There are downsides to both.

wait on lock:
Currently we only have a single background thread to do these kinds of tasks,
it is single threaded through each task.  So if you wait on lock, then every
background task waits until that can get finished.  Stuff like checkpoints,
and other space reclamation tasks will wait.  The queue may grow and become
unmanageable.

bump retry:
Could lead to just cpu spinning and no actual guarantee of getting the resource
as it could keep getting unlucky and lose a cycle schedule when the resource
is actually available.

Bigger feature projects could do resolve these issues.

1) Enhance the background daemon schedule utility to support more than one
   thread.  
   o I think it is important that we don't just randomly grow the number
     of background threads, as we already have complaints about the one
     background thread per db.  But it would be fine to short term allocate 
     a thread and then destroy it.
   o not sure exactly what is best, but in this case it would be nice to 
     be able to REQUE the post commit task to the daemon saying add it to the
     WAIT allowed queue.  At that point maybe the daemon starts a new thread
     for each one or maybe for each N or something else.  Maybe it is 
     configurable.
   o There are already other problems for which this may be the solution:

     1) As number of cores/cpu's grow it becomes obvious that one background
        thread to N possible concurrent user thread which could each generate
        work is not correct.
     2) There are some tasks that are even more critical than others that could
        benefit from better priority.  Things like checkpoint may want their
        own thread rather than share with others.


2) The problem with space being lost "forever" (which really means until an
   offline compress, in part stems from the row format of the overflow pieces.

   o we could change the overflow pieces to have back pointers to the main
     head page which would make it much easier to figure out a stranded piece
     during online compress.

   o We could write some sort of brute force search which could come up with
     the stranded pieces and call reclaim on them.  It is actually not that
     hard of code to write if one doesn't worry about memory.  It would go
     something like - probably many ways to optimize it. :

     for (every page at raw store level)
     {
         if (main page)
         {
             for every row on page
             {
                 if (has an overflow piece)
                     add main row handle to main hash table, and overflow handle
             }
         }

         if (overflow page)
         {
             for every row on page
             {
                 add main row handle, and next overflow handle
             }
         }

     }

     for (every row in main hash table)
     {
         delete complete overflow chain from overflow hash table
     }

     I believe this leaves the disconnected chains in the overflow hash table,
     with some graph algorithm to determine the head of the chains.

     The above works for X lock on the whole table, I am not sure if row locking
     works.  If you have to get X lock then an offline compress is not that much
     worse which is why it never got implemented.

3) This post commit work is all still really at the first implmentation level,
   when most choices were what was simple and mostly work.  Not much has 
   happened since then.  Also in first implementation big CLOB's and BLOB's were
   not even implemented so the downside of losing some stuff was not much.

   As Kathey pointed out I think there are holes in a lot of the post commit
   stuff where a crash can lose them.  It seems the only way to make sure we
   don't lose them is to somehow store them transactionally in the db, but at
   what cost?  We could implement some sort of internal table, and then store
   rows with the post commit work.

Personally I think 1 and 2 are better paths than 3.  It would be interesting
to know what other db's do.  I am pretter sure postgress for instance rely's
on scheduling a "scrubber" to do a lot of what we do in post commit.  I would
rather see us make changes that would allow online compress table to handle
these overflow losses and then enhance the post commit deamon to better handle
them asap without ever needing the compress table.



> Space may not be reclaimed if  row locks are not available after three retries 
> -------------------------------------------------------------------------------
>
>                 Key: DERBY-4055
>                 URL: https://issues.apache.org/jira/browse/DERBY-4055
>             Project: Derby
>          Issue Type: Bug
>          Components: Store
>    Affects Versions: 10.1.3.1, 10.2.2.0, 10.3.3.0, 10.4.2.0, 10.5.0.0
>            Reporter: Kathey Marsden
>         Attachments: derby.log.T_RawStoreFactoryWithAssert
>
>
> In a multithreaded clob update where the same row is being updated, space will not be
reclaimed.  The offending code is in ReclaimSpaceHelper:
> 	RecordHandle headRecord = work.getHeadRowHandle();
> 		if (!container_rlock.lockRecordForWrite(
>                 tran, headRecord, false /* not insert */, false /* nowait */))
> 		{
> 			// cannot get the row lock, retry
> 			tran.abort();
> 			if (work.incrAttempts() < 3)
>             {
> 				return Serviceable.REQUEUE;
>             }
> 			else
>             {
>                 // If code gets here, the space will be lost forever, and
>                 // can only be reclaimed by a full offline compress of the
>                 // table/index.
>                 if (SanityManager.DEBUG)
>                 {
>                     if (SanityManager.DEBUG_ON(DaemonService.DaemonTrace))
>                     {
>                         SanityManager.DEBUG(
>                             DaemonService.DaemonTrace, 
>                             "  gave up after 3 tries to get row lock " + 
>                             work);
>                     }
>                 }
> 				return Serviceable.DONE;
>             }
> 		}
> If we cannot get the lock after three tries we give up.  The reproduction for this issue
is in the test store.ClobReclamationTest.xtestMultiThreadUpdateSingleRow().
> This issue also used to reference the code below and has some references to trying to
get a reproduction for that issue, but that work has moved to DERBY-4054.  Please see DERBY-4054
for work on the container lock issue.
> ContainerHandle containerHdl = 
> 			openContainerNW(tran, container_rlock, work.getContainerId());
> 		if (containerHdl == null)
> 		{
> 			tran.abort();
> 			if (SanityManager.DEBUG)
>             {
>                 if (SanityManager.DEBUG_ON(DaemonService.DaemonTrace))
>                 {
>                     SanityManager.DEBUG(
>                         DaemonService.DaemonTrace, " aborted " + work + 
>                         " because container is locked or dropped");
>                 }
>             }
> 			if (work.incrAttempts() < 3) // retry this for serveral times
>             {
> 				return Serviceable.REQUEUE;
>             }
> 			else
>             {
>                 // If code gets here, the space will be lost forever, and
>                 // can only be reclaimed by a full offline compress of the
>                 // table/index.
>                 if (SanityManager.DEBUG)
>                 {
>                     if (SanityManager.DEBUG_ON(DaemonService.DaemonTrace))
>                     {
>                         SanityManager.DEBUG(
>                             DaemonService.DaemonTrace, 
>                             "  gave up after 3 tries to get container lock " + 
>                             work);
>                     }
>                 }
> 				return Serviceable.DONE;
>             }
> 		}	

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message