db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kristian Waagan (JIRA)" <j...@apache.org>
Subject [jira] Updated: (DERBY-4771) Continue investigation of automatic creation/update of index statistics
Date Tue, 07 Dec 2010 14:57:09 GMT

     [ https://issues.apache.org/jira/browse/DERBY-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Kristian Waagan updated DERBY-4771:

    Attachment: derby-4771-2h-prototype_lcc_code_dump.diff

Attaching untested patch 2h, which addresses Dag's comments from 02/Dec/10 11:31 AM.

(Q = question, C = comment, A = action)

A   java/engine/org/apache/derby/iapi/services/daemon/IndexStatisticsDaemon.java

* Various questions here:
  Q: How is this safe? (errors not reported?) explain!
  C: I guess it isn't totally safe. That said, except for writing statistics
     entries into a system table, the daemon is only reading data. The basic
     principle for the daemon when it comes to error handling is to ditch the
     current unit of work if there in an error and continue with the next one.
     It doesn't really care if the operation fails, because an update will be
     scheduled again. The daemon is currently separating between expected and
     unexpected errors. The former will simple be ignored, as they can arise
     for various reasons during "normal operation". One such error is failed
     invalidation because someone else is accessing the existing statistics.
     Unexpected errors are errors for which the daemon doesn't have specific
     code to handle. The best we can do here is to log it and continue if
  A: I have rewritten the logging code, such that unexpected errors are always
     logged, even if logging is disabled.

  Q: What happens if schedule throws?
  C: It doesn't. If it did, it would have to be
     handled by the code calling it.
  A: I removed the throws clause.

  Q: Under which condition can work queue be full? Then what hapens?
  C: If work is scheduled at a higher rate than the daemon can generate new
     statistics. Would typically only happen if there are very large tables in
     the database.
     If the queue is full, the work simply won't be added. It will eventually
     be added when there is free space in the queue and a query compilation
     triggers the scheduling.
  A: I think the current default queue size is too small (5). I have
     increased to 20 for now (worst case is 19 comparisons to look for
     duplicates per query compilation).

A   java/engine/org/apache/derby/impl/services/daemon/IndexStatisticsDaemonImpl.java

* I removed LOG_HEADER (used in only one place now).

* Various questions:
  Q: What are "in all cases" exactly?
  C: Currently the daemon has to take locks when invalidating statements and
     dropping/adding statistics entries.

  Q: Not in daemon mode?
  C: Yes, in explicit mode it will take locks during scanning as well to stop
     another transaction to drop the table/index.

  Q: Or is the first sentence not true: "if it needs to take locks it will give up immediately
if the locks cannot be obtained"
  C: This is true as far as there is an easy way to do it. I think the
  A: I rewrote the comment somewhat, and added a description of the flow.

* Removed comment. I think I added it when considering very large tables, but I
  don't understand it myself any more...

* Rewrote loop slightly to synchronize on "queue" when checking "daemonDisabled".

* Good!

* "daemonLCC" will be null the first time run is started. Since the daemon
  might not run at all, I decided to create the database connection lazily.
  I have now coded under the assumption that the daemon won't lose the
  connection to the database unless the database is shut down (for whatever
  reason). This basically consisted of removing parts of the comment for

* I added final to both.

* Yes, but not at this level. I thought it would be nice to have a name for the
  transaction to identify it in the transaction table.
  I think some new methods must be added to be able to name a transaction at
  this level, so I'm not sure if it is worth the trouble.
  I'm keeping the TODO for now, might remove it in the next iteration.

* Whether or not to remove the tracing code depends on the feedback received
  during testing. If people find it useful, it can be improved a little and
  kept, but if not it should be removed.
  I envision this happening closer to when the target release is cut.

* Added another sentence to the JavaDoc, stating that the table descriptor
  (unit of work) will be discarded if it turns out to be invalid when the
  statistics update is about to happen. I expect the most common error is that
  the table has been dropped since the work for it was scheduled.

* It isn't needed now. Since the interface is internal, and there is only one
  implementation of it, I suppose the best action to take now is to remove it.
  We can introduce it again later, and then probably in a shape more like you
  have described. It feels a bit odd to say in the JavaDoc that scheduling
  requests may be denied, and not have a way to learn if it happened or not...

* Added synchronization for runningThread in the finally-block.
  The current code will let the thread die and then create a new one on the
  next update request. I considered adding a sleep before letting the thread
  die, in case a new request would come in quickly.


* Do you mean we should call TransactionResourceImpl#handleException explicitly
  I think the comment meant to say that the daemon will be disabled elsewhere.
  I'll address this issue in the next iteration.

* Added simple shutdown mechanism if there are too many consecutive errors.

* Currently the queue is cleared only when the daemon is shut down.

* Here's another way it can happen. Parts of the trace may be familiar to you ;)
        at org.apache.derby.iapi.services.context.ContextManager.checkInterrupt(ContextManager.java:437)
        at org.apache.derby.iapi.services.context.ContextManager.getContext(ContextManager.java:155)
        at org.apache.derby.iapi.services.context.ContextService.getContextOrNull(ContextService.java:249)
        at org.apache.derby.iapi.util.InterruptStatus.setInterrupted(InterruptStatus.java:71)
        at org.apache.derby.iapi.util.InterruptStatus.noteAndClearInterrupt(InterruptStatus.java:112)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.recoverContainerAfterInterrupt(RAFContainer4.java:807)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:366)
        at org.apache.derby.impl.store.raw.data.RAFContainer4.readPage(RAFContainer4.java:246)
        at org.apache.derby.impl.store.raw.data.CachedPage.readPage(CachedPage.java:671)
        at org.apache.derby.impl.store.raw.data.CachedPage.setIdentity(CachedPage.java:190)
        at org.apache.derby.impl.services.cache.ConcurrentCache.find(ConcurrentCache.java:295)
        at org.apache.derby.impl.store.raw.data.FileContainer.getUserPage(FileContainer.java:2530)
        at org.apache.derby.impl.store.raw.data.FileContainer.getPage(FileContainer.java:2580)
        at org.apache.derby.impl.store.raw.data.BaseContainerHandle.getPage(BaseContainerHandle.java:319)
        at org.apache.derby.impl.store.access.btree.ControlRow.get(ControlRow.java:833)
        at org.apache.derby.impl.store.access.btree.ControlRow.get(ControlRow.java:820)
        at org.apache.derby.impl.store.access.btree.ControlRow.getRightSibling(ControlRow.java:531)
        at org.apache.derby.impl.store.access.btree.BTreeScan.positionAtNextPage(BTreeScan.java:493)
        at org.apache.derby.impl.store.access.btree.BTreeForwardScan.fetchRows(BTreeForwardScan.java:464)
        at org.apache.derby.impl.store.access.btree.BTreeScan.fetchNextGroup(BTreeScan.java:1596)
        at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl$KeyComparator.fetchRows(IndexStatisticsDaemonImpl.java:1016)
        at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.updateIndexStatsMinion(IndexStatisticsDaemonImpl.java:432)
        at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.updateAllIndexStats(IndexStatisticsDaemonImpl.java:329)
        at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.generateStatistics(IndexStatisticsDaemonImpl.java:291)
        at org.apache.derby.impl.services.daemon.IndexStatisticsDaemonImpl.run(IndexStatisticsDaemonImpl.java:650)
        at java.lang.Thread.run(Thread.java:662)

* I think I meant to retry the whole operation, that is to add the unit of work
  to the end of the queue or something. The way it is now, it will retry to get
  the locks, but if that fails it will discard the unit of work.
  Note that there is retry logic on several levels here.
  Since a new unit of work will be scheduled the next time a query using the
  index(es) is compiled, I'll delete the TODO and keep the code as it is.
  If this strategy turns out to be inadequate, we can fix it later.

* Fixed.

* Updated comment (and some formatting). 

Running tests.
I asked for opinions on three issues, and one issue will have to be addressed
in the next iteration (error handling).

I will now start splitting up the patch by creating a master JIRA with some subtasks.
I think I also need to write a wiki-page describing the current properties controlling the
feature, including the debug properties (for tuning during the development/testing phase).

> Continue investigation of automatic creation/update of index statistics
> -----------------------------------------------------------------------
>                 Key: DERBY-4771
>                 URL: https://issues.apache.org/jira/browse/DERBY-4771
>             Project: Derby
>          Issue Type: Task
>          Components: SQL, Store
>    Affects Versions:
>            Reporter: Kristian Waagan
>            Assignee: Kristian Waagan
>         Attachments: autoindexstats.html, derby-4771-1a-prototype_code_dump.diff, derby-4771-1a-prototype_code_dump.stat,
derby-4771-1b-prototype_code_dump.diff, derby-4771-2a-prototype_lcc_code_dump.diff, derby-4771-2b-prototype_lcc_code_dump.diff,
derby-4771-2c-prototype_lcc_code_dump.diff, derby-4771-2d-prototype_lcc_code_dump.diff, DERBY-4771-2e-prototype.rar,
derby-4771-2e-prototype_lcc_code_dump.diff, Derby-4771-2f-AutomaticIndexStatisticsTest_wondows7.rar,
derby-4771-2f-prototype_lcc_code_dump-WORK-IN-PROGRESS.diff, derby-4771-2f-prototype_lcc_code_dump.diff,
derby-4771-2g-prototype_lcc_code_dump.diff, derby-4771-2h-prototype_lcc_code_dump.diff, derby.log,
error-stacktrace.out, rjall.out, rjall.out, rjall.out, rjall.rar, rjone.out
> Work was started to improve Derby's handling of index statistics. This issue tracks further
discussion and work for this task.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message