flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Santhosh Chandrasekaran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3185) Corrupt event found. Please run File Channel Integrity tool
Date Wed, 11 Oct 2017 13:23:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16200265#comment-16200265
] 

Santhosh Chandrasekaran commented on FLUME-3185:
------------------------------------------------

We tried executing FC Integrity tool. It fixes the Corrupt Event.. But this occurs frequently
in PRODUCTION. corruption event removed file and original file (.bak) are of same size

/apps/osp/apache-flume-1.7.0/bin/flume-ng tool FCINTEGRITYTOOL  -l /appsdata/osp/elastic/datastore/flume/esu1l772/ems/d2cpaf_prod23/data/chn-file-xml-dis-log/
-Xms2g -Xmx2g
Warning: No configuration directory set! Use --conf <dir> to override.
Info: Including Hive libraries found via () for Hive access
FLUME_CLASSPATH =  /apps/osp/apache-flume-1.7.0/plugins.d/flume_processors/libext/*:/apps/osp/apache-flume-1.7.0/plugins.d/flume_processors/lib/*:/apps/osp/apache-flume-1.7.0/lib/*:/lib/*
+ exec /apps/osp/jdk1.8.0_131/bin/java -Xmx20m -Xms2g -Xmx2g -cp '/apps/osp/apache-flume-1.7.0/plugins.d/flume_processors/libext/*:/apps/osp/apache-flume-1.7.0/plugins.d/flume_processors/lib/*:/apps/osp/apache-flume-1.7.0/lib/*:/lib/*'
-Djava.library.path= org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /appsdata/osp/elastic/datastore/flume/esu1l772/ems/d2cpaf_prod23/data/chn-file-xml-dis-log/
log4j:WARN No appenders could be found for logger (org.apache.flume.tools.FileChannelIntegrityTool).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
---------- Summary --------------------
Number of Events in the Channel = 14407
Number of Put Events Processed = 7280
Number of Valid Put Events = 7280
Number of Invalid Put Events = 0
Number of Put Events that threw Exception during validation = 0
Number of Corrupt Events = 1
---------------------------------------


FC Integrity tool logs:- (Offset is 0)

10 Oct 2017 20:03:39,225 WARN  [main] (org.apache.flume.tools.FileChannelIntegrityTool.run:136)
 - Corruption found in /appsdata/osp/elastic/datastore/flume/esu1l772/ems/prod_corr/log-98
at 0
10 Oct 2017 20:03:39,507 INFO  [main] (org.apache.flume.channel.file.LogFile$OperationRecordUpdater.markRecordAsNoop:437)
 - Marking event as 0 at 0 for file /appsdata/osp/elastic/datastore/flume/esu1l772/ems/prod_corr/log-98
10 Oct 2017 20:03:39,511 INFO  [main] (org.apache.flume.channel.file.LogFile$SequentialReader.next:683)
 - Encountered EOF at 86589353 in /appsdata/osp/elastic/datastore/flume/esu1l772/ems/prod_corr/log-98
~


> Corrupt event found. Please run File Channel Integrity tool
> -----------------------------------------------------------
>
>                 Key: FLUME-3185
>                 URL: https://issues.apache.org/jira/browse/FLUME-3185
>             Project: Flume
>          Issue Type: Bug
>          Components: File Channel
>    Affects Versions: 1.7.0
>         Environment: PRODUCTION
>            Reporter: Santhosh Chandrasekaran
>
> We get the below exception in PROD.
> 08 Oct 2017 15:46:09,988 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:158)
 - Unable to deliver event. Exception follows.
> org.apache.flume.ChannelException: Take failed due to IO error [channel=chn-file-xml-dis-log]
>         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:534)
>         at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)
>         at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)
>         at com.macys.daas.flume.sink.elasticsearch.CreateIndex.process(CreateIndex.java:150)
>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.IOException: Corrupt event found. Please run File Channel Integrity
tool.
>         at org.apache.flume.channel.file.Log.get(Log.java:616) 
>         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doTake(FileChannel.java:531)
>         ... 6 more
> Caused by: org.apache.flume.channel.file.CorruptEventException: Could not parse event
from data file.
>         at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:212)
>         at org.apache.flume.channel.file.LogFileV3$RandomReader.doGet(LogFileV3.java:303)
>         at org.apache.flume.channel.file.LogFile$RandomReader.get(LogFile.java:501)
>         at org.apache.flume.channel.file.Log.get(Log.java:612)
>         ... 7 more
> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
>  at com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:110)
>         at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:755)
>         at com.google.protobuf.CodedInputStream.isAtEnd(CodedInputStream.java:701)
>         at com.google.protobuf.CodedInputStream.readTag(CodedInputStream.java:99)
>         at org.apache.flume.channel.file.proto.ProtosFactory$Put.<init>(ProtosFactory.java:3979)
>         at org.apache.flume.channel.file.proto.ProtosFactory$Put.<init>(ProtosFactory.java:3943)
>         at org.apache.flume.channel.file.proto.ProtosFactory$Put$1.parsePartialFrom(ProtosFactory.java:4039)
>         at org.apache.flume.channel.file.proto.ProtosFactory$Put$1.parsePartialFrom(ProtosFactory.java:4034)
>         at com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:200)
>         at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:241)
>         at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
>         at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
>         at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
>         at org.apache.flume.channel.file.proto.ProtosFactory$Put.parseDelimitedFrom(ProtosFactory.java:4179)
>         at org.apache.flume.channel.file.Put.readProtos(Put.java:97)
>         at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:206)
>         ... 10 more
> 08 Oct 2017 15:46:14,866 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:158)
 - Unable to deliver event. Exception follows.
> java.lang.IllegalStateException: Log is closed
>         at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
>         at org.apache.flume.channel.file.Log.getFlumeEventQueue(Log.java:591)
>         at org.apache.flume.channel.file.FileChannel$FileBackedTransaction.<init>(FileChannel.java:442)
>         at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:359)
>         at org.apache.flume.channel.BasicChannelSemantics.getTransaction(BasicChannelSemantics.java:122)
>         at com.macys.daas.flume.sink.elasticsearch.CreateIndex.process(CreateIndex.java:136)
>         at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
>         at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
>         at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message