logging-log4j-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arkin Yetis <arkinye...@gmail.com>
Subject Re: Flume Appender failure due to filesystem issue
Date Sun, 06 Apr 2014 00:52:01 GMT
Yes, I was referring to the Flume Persistent Appender. I will open an
enhancement request in JIRA.In the meanwhile, we will consider using local
disk instead of a remote filesystem to decrease the likelihood of the issue
occurring.

Thanks,
- Arkin


On Fri, Apr 4, 2014 at 2:46 PM, Ralph Goers <ralph.goers@dslextreme.com>wrote:

> I'm assuming you are using the Flume Persistent Appender which uses
> Berkeley DB based on the logs below.  From the log it appears that Berkeley
> DB's file handle is stale and meets to be closed and reopened.  I haven't
> checked but Berkeley DB might have a setting for this, otherwise the Flume
> Persistent Manager would need to deal with this condition. That would be a
> big change as the Database object is currently immutable.
>
> Ralph
>
> On Apr 4, 2014, at 12:04 PM, Arkin Yetis <arkinyetis@gmail.com> wrote:
>
> > We use the Flume Appender. Our logging stopped after a certain point in
> > time and we noticed the exception at the end of this message in our
> > application logs. It looks like there was an issue with the filesystem.
> But
> > although the filesystem has recovered, the appender (or probably the
> > persistence mechanism it uses) was stuck in this state and it took an
> > application restart for it to continue logging. It does not look like
> there
> > is a recovery mechanism or if there is one it failed.
> > Would you like me to open a log4j JIRA ticket for this? Or is this
> > something that can be prevented by something simple you can share over
> > e-mail such as a certain configuration setting?
> >
> > Thanks,
> > - Arkin
> >
> > Exception stack is:
> > 1. Stale NFS file handle (java.io.IOException)
> >  java.io.RandomAccessFile:-2 (null)
> > 2. Environment invalid because of previous exception: (JE 5.0.73)
> > /app/logs/abs-workflow/flumeDir java.io.IOException: Stale NFS file
> handle
> > LOG_READ: IOException on read, log is likely invalid. Environment is
> > invalid and must be closed. fetchTarget of 0x542/0x4af13c parent IN=5 IN
> > class=com.sleepycat.je.tree.BIN lastFullVersion=0x543/0x62d6c5
> > lastLoggedVersion=0x543/0x62d6c5 parent.getDirty()=true state=0
> > (com.sleepycat.je.EnvironmentFailureException)
> >  com.sleepycat.je.log.FileManager:1883 (null)
> >
> >
> ********************************************************************************
> > Root Exception stack trace:java.io.IOException: Stale NFS file handle
> >    at java.io.RandomAccessFile.readBytes(Native Method)
> >    at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
> >    at
> >
> com.sleepycat.je.log.FileManager.readFromFileInternal(FileManager.java:1918)
> >    at
> com.sleepycat.je.log.FileManager.readFromFile(FileManager.java:1869)
> >    at
> com.sleepycat.je.log.FileManager.readFromFile(FileManager.java:1807)
> >    at com.sleepycat.je.log.FileSource.getBytes(FileSource.java:56)
> >    at
> >
> com.sleepycat.je.log.LogManager.getLogEntryFromLogSource(LogManager.java:919)
> >    at com.sleepycat.je.log.LogManager.getLogEntry(LogManager.java:848)
> >    at
> >
> com.sleepycat.je.log.LogManager.getLogEntryAllowInvisibleAtRecovery(LogManager.java:809)
> >    at com.sleepycat.je.tree.IN.fetchTarget(IN.java:1412)
> >    at com.sleepycat.je.tree.BIN.fetchTarget(BIN.java:1251)
> >    at com.sleepycat.je.dbi.CursorImpl.fetchCurrent(CursorImpl.java:2261)
> >    at
> >
> com.sleepycat.je.dbi.CursorImpl.getCurrentAlreadyLatched(CursorImpl.java:1466)
> >    at com.sleepycat.je.dbi.CursorImpl.getNext(CursorImpl.java:1593)
> >    at
> >
> com.sleepycat.je.cleaner.UtilizationProfile.getObsoleteDetail(UtilizationProfile.java:632)
> >    at
> >
> com.sleepycat.je.cleaner.FileProcessor.processFile(FileProcessor.java:439)
> >    at
> > com.sleepycat.je.cleaner.FileProcessor.doClean(FileProcessor.java:289)
> >    at
> > com.sleepycat.je.cleaner.FileProcessor.onWakeup(FileProcessor.java:148)
> >    at com.sleepycat.je.utilint.DaemonThread.run(DaemonThread.java:163)
> >    at java.lang.Thread.run(Thread.java:662)
> >
> ********************************************************************************
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message