Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: <derby-dev@db.apache.org>
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=sbcglobal.net;
  h=DKIM-Signature:Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding;
  b=yCyMTk0r7SxZ9ESWtq4WhVSCjbxLcHIEwqYLkX1iDywuYr1Yile7ca5iMvQLNKO7pU0J/LilSBHVjDsOJqqDFqyZZ2CkV8in3L7BHWc0pRVOdBY0TQFGb31RasiDhG+4RvyNX871vk5y7KobYhlSUrSDfwDer2M5/tPSWpiBb74=
  ;
Message-ID: <4C5C3320.9090604@sbcglobal.net>
Date: Fri, 06 Aug 2010 09:06:56 -0700
From: Mike Matrigali <mikem_app@sbcglobal.net>
User-Agent: Thunderbird 2.0.0.24 (Windows/20100228)
MIME-Version: 1.0
To: derby-dev@db.apache.org
Subject: Re: Transaction contexts and log flushing
References: <4C581C29.5060304@oracle.com> <4C5851D6.3020002@sbcglobal.net>
 <4C586E74.8070009@oracle.com> <4C58849C.4070701@sbcglobal.net>
 <4C5BE2C3.2070301@oracle.com>
In-Reply-To: <4C5BE2C3.2070301@oracle.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Kristian Waagan wrote:
> The new stats are stored in the daemon. There is logic in 
> GenericStatement (at the very end of prepMinion) to go to the daemon to 
> fetch any new stats if they exists. Although all new stats will be 
> written at once, the decision on whether to go to the daemon to ask for 
> new stats is taken based on the statistics needed by the query being 
> compiled.
> 
> Another issue here is whether doing these checks only at compile time is 
> good enough. May the system reach a steady state where the stats are 
> becoming outdated and no queries are (re)compiled?
> 
Ok, now I think I understand the async and how you are using the user 
thread.  I had not considered using subsequent executions to do the
system catalog update.  I think it would be a little less complicated
if we could figure out a way to just let the daemon do the update, but
previous attempts did run into problems.  I can definitely see cases
where we may never compile again in a server session, and it would be
sad to do all the work to create the stats and never get them written.

I think the most important case to handle is where no stats exist at 
all.  For a table of any significant size stats should not become
outdated, and determining if they are outdated may take just as much
work as compiling them again.  For derby the only stats that we are
talking about here is the cardinality - basically how many duplicates
per key.  static histogram stats are not kept, instead at compile time
the actual indexes are used for histogram based costing of key ranges.

maybe longer term we can invalidate based on large changes in number
of rows, or just date that last statistics were gathered.  And really
longer term we could invalidate if we figure out during execution that
compile time estimate was wrong.