Mailing-List: contact derby-dev-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Derby Development" <derby-dev@db.apache.org>
Received-SPF: neutral (hermes.apache.org: local policy)
Message-ID: <428E7350.5060202@sbcglobal.net>
Date: Fri, 20 May 2005 16:31:28 -0700
From: Mike Matrigali <mikem_app@sbcglobal.net>
User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206)
MIME-Version: 1.0
To: Derby Development <derby-dev@db.apache.org>
Subject: Re: [PATCH] increasinng the possible log file id's from 2^22 -1 to
 2^33 -1 (derby-101)
References: <428956A5.8050806@gmail.com> <428A3C42.6020601@sbcglobal.net>
 <428D2517.8000402@gmail.com> <428E37D9.8080809@gmail.com>
In-Reply-To: <428E37D9.8080809@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

This seems like a reasonable 1st step, that would solve current
upgrade issues.  I suggest you submit this patch, close the current
jira, and submit a new future improvement jira noting what would
have to be done to eek out the last few bits (or documenting it
somewhere appropriate in the code is also fine - just rather not see
you have learned lost).  Given that this
soft upgrade stuff is new I would rather be safe this time.

/mikem

Suresh Thalamati wrote:
> By looking at the code , it is kind of tricky to handle upgrade, because 
> users can crash in the middle of recovery
> and reboot again N times , which can generate new log for uncommitted 
> transaction and also all the constant that manipulates LSN
> are static now  and are used by static routines in 
> org.apache.derby.impl.store.raw.log.LogCounter.java, like 
> :makeLogInstantAsLong().
> 
> To handle upgrade:
> 1)   all static methods  in LogCounter.java has to be replaced by  
> instance methods , so that
>    in the same JVM, one  can run a  database that is in softupgrade mode 
> and one that is new/upgraded..
> 2)  modify the code to update the version number on disk(log control 
> file )  to only when  the recovery is complete;
>      currently it is updated in the boot code itself.
> 3)  perform a checkpoint after recovery on upgrade, so that new log file 
> limits can be uses, As it will make
>    sure that no old log instant format will be in use.
> 
> Instead of  doing all of the above , I can reduce the limit to 2^31 -1 
> instead of 2^33-1 ;
> this way I would not need to change LogCounter.FILE_NUMBER_SHIFT  from 
> 32 (old value) to 30,
> which causes all upgrade issues.
> 
> with 2^31 -1 it will take approximately 68 years (2147483647/(24 * 60 * 
> 60 * 365))
> to hit the log file number limit  if a new log file  is generated every 
> second., instead of 272 years
> with 2^33-1.
> 
> If no one has objections , I would prefer to increase the log file 
> number limit to 2^31 -1 and
> avoid major upgrade issues. If any one strongly believes  it should be 
> 2^33 -1 ,  I will try
> to do the necessary upgrade changes.
> 
> Any comments/suggestions ?
> 
> Thanks
> -suresht
> 
> 
> 
> 
> Suresh Thalamati wrote:
> 
>> I just realized , this patch actaully break's hard upgrade also, if  
>> log needs to be replayed.
>> Becuase I extract the log file number and log file postion from the 
>> instant that is on disk ,
>> differently after my change.
>>
>> If  any one else on the list  is testing  upgrade is hitting this 
>> problem, please let me know.
>> I can post a patch to backout the portion that cause upgrade issues.
>>
>>
>> Thanks
>> -suresht
>>
>>
>> Mike Matrigali wrote:
>>
>>> I am looking at committing this.  The changes look good to me, but I
>>> believe there are upgrade issues to handle.
>>>
>>> For a hard upgrade either new or old databases are fine.
>>> For a soft upgrade I think there is a problem if the db generates enough
>>> log files to start using the new bits, and then the software is reverted
>>> to before the fix.
>>>
>>> Seems like using the bits needs to be somehow only enabled for hard
>>> upgrade.  It would be best if it was controlled just by hard upgrade,
>>> but if that is not possible then just doing it for databases created
>>> since this version would also work - but still leave problems with old
>>> hard upgraded databases.
>>>
>>> I have reviewed the code and am running tests.  I plan on committing
>>> this part of the fix and let you address upgrade issues with a follow on
>>> patch.
>>>
>>> Suresh Thalamati wrote:
>>>
>>>  
>>>
>>
>>
>>
> 
> 
> 
>