Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Andrew Jones <andrew+flume@andrew-jones.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: FileChannel error
Message-Id: <6068398F-BDF2-4AB9-9033-1A5378878F03@andrew-jones.com>
Date: Fri, 29 Mar 2013 18:02:40 +0000
To: user@flume.apache.org
Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\))

Hi,

I restarted my flume process, and I am now getting the following error =
in my logs:

29 Mar 2013 17:56:13,756 ERROR [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.LogFile$SequentialReader.next:493)  -
Encountered non op-record at 1357908629 3e in
/var/run/flume/file-channel/flume-log-sink-dev/data/log-883
29 Mar 2013 17:56:13,760 ERROR [lifecycleSupervisor-1-0]
(org.apache.flume.channel.file.Log.replay:410)  - Failed to initialize
Log on [channel=3Dchannel]
java.io.IOException: Unable to read next Transaction from log file
/local/flume/file-channel/flume-log-sink-dev/data/log-884 at offset
720893818
       at =
org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:5=
02)
       at =
org.apache.flume.channel.file.ReplayHandler.next(ReplayHandler.java:364)
       at =
org.apache.flume.channel.file.ReplayHandler.replayLog(ReplayHandler.java:2=
64)
       at org.apache.flume.channel.file.Log.doReplay(Log.java:435)
       at org.apache.flume.channel.file.Log.replay(Log.java:382)
       at =
org.apache.flume.channel.file.FileChannel.start(FileChannel.java:303)
       at =
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(Lifecyc=
leSupervisor.java:236)
       at =
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
       at =
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)=

       at =
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
       at =
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acces=
s$201(ScheduledThreadPoolExecutor.java:165)
       at =
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(S=
cheduledThreadPoolExecutor.java:267)
       at =
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1110)
       at =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java=
:603)
       at java.lang.Thread.run(Thread.java:679)
Caused by: com.google.protobuf.InvalidProtocolBufferException: While
parsing a protocol message, the input ended unexpectedly in the middle
of a field.  This could mean either than the input has been truncated
or that an embedded message misreported its own length.
       at =
com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(Invali=
dProtocolBufferException.java:49)
       at =
com.google.protobuf.CodedInputStream.readRawVarint32(CodedInputStream.java=
:402)
       at =
com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(Abstrac=
tMessageLite.java:280)
       at =
com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMes=
sage.java:760)
       at =
com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(Abstrac=
tMessageLite.java:288)
       at =
com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMes=
sage.java:752)
       at =
org.apache.flume.channel.file.proto.ProtosFactory$TransactionEventFooter.p=
arseDelimitedFrom(ProtosFactory.java:4559)
       at =
org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(Transac=
tionEventRecord.java:203)
       at =
org.apache.flume.channel.file.LogFileV3$SequentialReader.doNext(LogFileV3.=
java:344)
       at =
org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:4=
98)
       ... 14 more

So it seems to be have corrupted the log somehow, and is unable to =
recover. How can I get past this, either by removing the offending =
transaction, or making it recover? I don't want to lose all the events =
in the log.

Using Flume 1.3.1, Avro source and HDFS sink.

Thanks,
Andrew=