Mailing-List: contact derby-user-help@db.apache.org; run by ezmlm
Precedence: bulk
Reply-To: "Derby Discussion" <derby-user@db.apache.org>
Received-SPF: neutral (herse.apache.org: local policy)
Message-ID: <455CAE58.1010104@sbcglobal.net>
Date: Thu, 16 Nov 2006 10:30:48 -0800
From: Mike Matrigali <mikem_app@sbcglobal.net>
Reply-To: mikem_app@sbcglobal.net
User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206)
MIME-Version: 1.0
To: Derby Discussion <derby-user@db.apache.org>
Subject: Re: java.sql.SQLException: Log Corrupted, has invalid data in the
 log stream.
References: 
 <2526D42D65F597428BFF422D6AB79E9E0108FB9D@NA1000EXM01.na.ds.monsanto.com>
In-Reply-To: 
 <2526D42D65F597428BFF422D6AB79E9E0108FB9D@NA1000EXM01.na.ds.monsanto.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Deleting the log files by hand is an almost guaranteed way to corrupt
the data in the database:  uncommitted transactions can't be aborted,
data not flushed from cache will never make it to disk in case of
process crash, machine crash, unclean shutdown

Unless all data from the database cache
was synced to disk and all transactions were committed before the
last shutdown then deleting the log files basically gives you access
to an inconsistent db.  The fact that you are running through redo
recovery and getting an error makes it likely that the last shutdown
of the db was not clean, and thus the data on disk is missing updates
described in the log file to make the db consistent.  .  You can run the
consistency checker to identify some of the possible corruptsions
resulting from removing the log files.  The only safe thing to do
in case of log corruption is to go to a backup.

In my support experience if we ever need
to delete log files, we usually recommend unloading all the data
from the db and loading it to a new db to guarantee no ongoing
corruption.

I assume you can't reproduce this yourself.  With these kinds of
cases it is probably best to log a single tracking JIRA and load
as much info as possible into it.  Maybe someone can see a pattern.

This is the first I have heard of this kind of problem happening
so often.  The only cases I can think of are cases where somehow
there are 2 uncoordinated writers to the log file.  This can easily
happen with JVM's prior to 1.4 on non-windows OS's.

As you are seeing this frequently, some questions:
1) is it possible that the same database is being accessed from within
    the same JVM but from 2 different class loaders (ie. DERBY-700)?
2) Is the database located on a disk which can be accessed by 2
    different machines, and if so is it possible 2 machines are looking
    at the db using derby?  Note it does not even take 2 updating
    accesses to cause problems, just the act of booting a db can
    cause writes to the log file.
3) are you running with  derby.system.durability=test?
4) Are the disks where the log files are written set to cache writes,
    or do they do proper disk sync when requested.  Are they scsi or
    ide?
5) can you post any of the data or the log files, obviously if the
    data is private in any way you won't be able to put it in JIRA.
6) Is the log error in derby.log always exactly the same?  If not
it may help to post multiple derby.log's, again maybe someone will
see a pattern.  For instance is the nested stack
in derby.log always say - it would probably be really interesting if
the number was always 768:
java.lang.ClassNotFoundException: ERROR XBM0U: No class was registered 
for identifier 768.

Can you provide a listing of all the files in the log directory with
size and last modified time for some of log corruption cases.  More
than one case may show a pattern.  For instance is the nested stack
in derby.log always say:
java.lang.ClassNotFoundException: ERROR XBM0U: No class was registered 
for identifier 768.
What is happening in the one derby.log you posted here is that every 
derby log record has a format
id, so the process of reading the log file is to read the format id
and use that format id to instantiate the right class which represents
the subsequent log record.  In this case it read 768 - but the
highest format id that derby has is 464 (see MAX_ID_2 in
opensource/java/engine/org/apache/derby/iapi/services/io/StoredFormatIds.java).
Looking at the state of the particular log.dat file that caused this
error may lend some clues (ie. Is this the last record? Is there any 
pattern at this point, is this the last log file?).


Just to get an idea of scope of your app.  Are those 1000 users all
using the same database, on the same machine?
If not, how many are likely to be using the same db on the same machine?
If it is one machine is it a multi-processor, if so how many processors?


BALDWIN, ALAN J [AG-Contractor/1000] wrote:
> I have not *noticed* any inconsistencies after deleting these files,
> however, that doesn't mean there aren't any.  It just means they haven't
> been reported.  
> 
> I have never run the compress table routines.  Is that something that
> can be done programmatically at runtime?  
> 
> The corruption is noticed when we try to start the app and connect to
> the database.  It won't let us connect.  Also, as expected, we cannot
> connect using tools like Cloudscape Workbench either once this state is
> reached.
> 
> I can't think of anything else that is relevant at this point.  It seems
> to be random, and it does not seem to be related to users killing the
> app halfway through a transaction or anything like that.
> 
> -Alan-
> 
> -----Original Message-----
> From: Mayuresh Nirhali [mailto:Mayuresh.Nirhali@Sun.COM] 
> Sent: Thursday, November 16, 2006 6:04 AM
> To: Derby Discussion
> Subject: Re: java.sql.SQLException: Log Corrupted, has invalid data in
> the log stream.
> 
> Alan,
> 
> As you are aware, it is not recommended to delete the log fils manually.
> The logCorruptedException has another nestedException and it is because 
> the value read from the log records is not matched with required 
> appropriate data.
> But, Did you observe any sort of inconsistencies in your database when 
> you boot again after you delete the files. ?
> 
> How often do you (or did you ever) run the compress table routines ??
> 
> Do you see the log corrupted when you boot the db or during some 
> transactions ??
> 
> any more information which you think is relevant ??
> 
> Mayuresh
> 
> 
> BALDWIN, ALAN J [AG-Contractor/1000] wrote:
> 
> 
>>More info:
>>
>>Windows XP
>>
>>JVM 1.5 and JVM 1.4.2 
>>
>>
>>
>>I don't see any exceptions being thrown in our application before any
> 
> of the
> 
>>corruption cases that would indicate that something in the app had gone
>>awry.  
>>
>>
>>
>>Also, attached is my derby.log file.
>>
>>
>>
>>-Alan-
>>
>>
>>
>>
>>
>>-----Original Message-----
>>From: BALDWIN, ALAN J [AG-Contractor/1000] 
>>Sent: Thursday, November 16, 2006 9:42 AM
>>To: derby-user@db.apache.org
>>Subject: java.sql.SQLException: Log Corrupted, has invalid data in the
> 
> log
> 
>>stream.
>>
>>
>>
>>Hi all,
>>
>>We are having log corruption issues with Derby inside of a java swing
>>application.
>>
>>Here is the error:
>>
>>java.sql.SQLException: Log Corrupted, has invalid data in the log
> 
> stream.
> 
>>We have about 1000 users using this software, and I would guess that at
> 
> any
> 
>>given time we see 4-5 with log corruption.  Deleting "logXXX.dat" files
> 
> from
> 
>>within the log folder fixes the problem for the time being, but I would
> 
> like
> 
>>to get at the root of the problem.  Is this safe to do as an interim
>>solution?  I assume these are transaction logs, so it doesn't seem like
> 
> a
> 
>>good idea to delete them, but that is the only thing that seems to
> 
> work.
> 
>>Also, anecdotally, this seems to be more prevalent in version 10.2
> 
> since we
> 
>>upgraded from 10.1.3 a few weeks ago.
>>
>>I'm also going to dig around on JIRA to seem if I come up with
> 
> anything.
> 
>>Any ideas?
>>
>>Thanks,
>>
>> 
>>
> 
> 
> 
> ---------------------------------------------------------------------------------------------------------
> This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.
> 
> 
> All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware". Monsanto accepts no liability for any damage caused by any such code transmitted by or accompanying this e-mail or any attachment.
> ---------------------------------------------------------------------------------------------------------
> 
> 
>