db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Hillegas <Richard.Hille...@Sun.COM>
Subject Re: regarding derby.log being overwritten on startup. [was Re: [jira] Commented: (DERBY-515) Network Server should log server start and shutdown time to derby.log]
Date Mon, 22 Aug 2005 23:03:51 GMT
Derby still has an error log reading VTI: ErrorLogReader. It shows how a 
reasonable engineer, in a pre-xml world, would approach the problem of 
parsing a log file and presenting it in tabular form.

I retract my hasty suggestion that it would be easy to convert Derby 
logs to a more structured format. Although BasicGetLogHeader 
standardizes the leading fields of a log record, often the trailing 
fields are tastier...and today  these come to the logging subsystem as a 
single, unstructured string.

-Rick

Mike Matrigali wrote:

> I believe that the current log file format has been written to
> be easy to parse - but not as easy as looking for a single
> delimiter.  Remember that the log file can contain arbitrary
> text - so not sure if a single delimiter will work.  The worst
> case is log statement text which will print out values of fields,
> which can contain anything and are not under our control.
> At one point there was a tool that could take
> the log file and present it like a read only table, similar to
> the lock table.  Adding some delimiters might make it easier.
>
> I also like having the log file at least somewhat user readable.
>
> Having said that I believe that if someone really wanted the XML
> version, an alternate error logging module implementation could
> be provided which could either produce XML instead of text or
> could provide 2 different files.  If someone wanted to write a
> cool tool that parsed and presented the error log file, or needed
> xml to adapt it to an existing tool then I could see XML form
> being more important.
>
> There was hope to be able to be able to do thing like take the out
> of log statement text and then turn around and use that to drive
> a testing framework which could execute those statments.
>
> Danesh wrote:
>
>> I agree with going the delimited format route. Having the database
>> write out XML would probably impact preformance quite a bit. I also
>> wonder what would happen if the database were to die (power failure or
>> the like) and the XML file ended up invalid due to it missing those
>> crucial few closing tags. To simplify things and satisfy both parties
>> someone could write a log conversion tool i.e. DEL -> XML.
>>
>> ~Danesh
>>
>> On 8/19/05, David Van Couvering <David.Vancouvering@sun.com> wrote:
>>
>>> Please, God, no :).  Machines like reading XML but people (at least
>>> certain people) find it a bit horrific.
>>>
>>> What I have seen done in the past is using a simple delimiter 
>>> format, e.g.
>>>
>>> |Fri Aug 19 14:06:31 PST
>>> 2005|org.apache.derby.engine.impl.jdbc.EmbedConnection|Thread
>>> 275|Invalid user, please try again| | |
>>>
>>> then you can write tools that slurp this right into Excel, load into a
>>> database, etc.  We did this with an app I worked on at Sybase and it 
>>> was
>>> quite successful.
>>>
>>> David
>>>
>>> Rick Hillegas wrote:
>>>
>>>
>>>> At the risk of overloading this topic: While we're talking about
>>>> improvements to our error logging, I'd like to see us emit more
>>>> structured logs. This is one of the things that xml is actually good
>>>> for. It would be pretty easy to turn our log records into xml entries.
>>>> This would make it easy to write or use off-the-shelf tools for
>>>> viewing the logs and for filtering signal out of the noise.
>>>>
>>>> Cheers,
>>>> -Rick
>>>>
>>>> David Van Couvering wrote:
>>>>
>>>>
>>>>> Well, the most common solution is log rotation.  On restart, you
>>>>> rename the old log to derby.log.1 and then derby.log.2 and so on.
>>>>> After <n> log files, you start over again at derby.log.1.
>>>>> This is also useful for long-running systems so you don't run out of
>>>>> disk space because of your error logs.  When the error log gets a
>>>>> configurable X MBs, the system copies the log to derby.log.1,
>>>>> derby.log.2, etc.,  and starts a fresh log.  After N log files it
>>>>> starts back at 1 and overwrites the old log file.  Similar but
>>>>> different from above, but both are great to reduce administrative
>>>>> overhead for the user.
>>>>>
>>>>> David
>>>>>
>>>>> Sunitha Kambhampati wrote:
>>>>>
>>>>>
>>>>>> Øystein Grøvlen wrote:
>>>>>>
>>>>>>
>>>>>>>>>>>> "KM" == Kathey Marsden <kmarsdenderby@sbcglobal.net>
writes:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>   KM> Øystein Grøvlen (JIRA) wrote:
>>>>>>>   >> [    >> This would be even more helpful if the
derby.log file
>>>>>>> was not overwritten on the next startup.  (Probably a separate

>>>>>>> issue).
>>>>>>>   >>    >>    >>    KM> The derby.infolog.append
property is
>>>>>>> helpful in this regard.
>>>>>>>
>>>>>>>   KM>
>>>>>>> http://db.apache.org/derby/docs/10.1/tuning/rtunproper13217.html
>>>>>>>
>>>>>>> Thanks, Kathey, that is really what I need.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> One concern I have is that by default, the derby.log gets
>>>>>> overwritten on subsequent boot. From my experience with customers,
>>>>>> whenever there is an issue, I always end up telling them to add this
>>>>>> property almost every time to see the debug logs. It is very common
>>>>>> to see users reboot the system if they hit an error and in this
>>>>>> case, it is probable that important relevant information in the
>>>>>> derby.log is lost because it gets overwritten ( ex - recovery issues
>>>>>> , deadlock etc ).
>>>>>>
>>>>>> Maybe we could do something smart for derby.log , a balance between
>>>>>> keeping history across boots and disk space taken by the derby.logs.
>>>>>>
>>>>>> Any comments ?
>>>>>>
>>>>>> Sunitha.
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>


Mime
View raw message