Return-Path: X-Original-To: apmail-flume-user-archive@www.apache.org Delivered-To: apmail-flume-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 3E9731739E for ; Mon, 3 Nov 2014 21:23:32 +0000 (UTC) Received: (qmail 77674 invoked by uid 500); 3 Nov 2014 21:23:31 -0000 Delivered-To: apmail-flume-user-archive@flume.apache.org Received: (qmail 77620 invoked by uid 500); 3 Nov 2014 21:23:31 -0000 Mailing-List: contact user-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flume.apache.org Delivered-To: mailing list user@flume.apache.org Received: (qmail 77610 invoked by uid 99); 3 Nov 2014 21:23:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2014 21:23:31 +0000 X-ASF-Spam-Status: No, hits=2.9 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [216.109.115.110] (HELO nm46-vm7.bullet.mail.bf1.yahoo.com) (216.109.115.110) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 03 Nov 2014 21:23:03 +0000 Received: from [66.196.81.173] by nm46.bullet.mail.bf1.yahoo.com with NNFMP; 03 Nov 2014 21:19:52 -0000 Received: from [98.139.215.230] by tm19.bullet.mail.bf1.yahoo.com with NNFMP; 03 Nov 2014 21:19:52 -0000 Received: from [127.0.0.1] by omp1070.mail.bf1.yahoo.com with NNFMP; 03 Nov 2014 21:19:52 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 309992.24077.bm@omp1070.mail.bf1.yahoo.com Received: (qmail 17429 invoked by uid 60001); 3 Nov 2014 21:19:52 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1415049592; bh=wFODil5W8HBQpxej/GAx0ajiOB9CRFUEov4n7kFGYLM=; h=References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=WskzyFFhaIw6rX8E/rzkZitmkz4yTwALPth9Pb560oLpC5XjbQFbwZenJ1omegvLElLbWBMl6JngyD9g8UcdiABD0NjeA6RYyWBC5VbsdBGK+EP++y5VmdHGhbZr4FgcWezxB41MtAAEyL1KqtEii32fg8b79dxR2+jownDeS+Y= X-YMail-OSG: 55NmQUcVM1nm4wUKIJpqpab9MOeZVV16eSSGyY3ymKpxVvq IeWkU7tZte0CyF_hSaHib3olchhgTSQaViMt.Rn0wv5Gko1EpzlOvFhOgbFz 4z_w6YmwurF3BRQwSBAJKVlG6rOGT.yns7_hiVX_ywZmR69igAhBKJCw_h5P QmF2Y5M_xUwWEK8omumbFldSevxQhTQpnAn3umi0ZLGT_tns5o7ALy3I5Oj1 intxWU9YyEuPPG3MuOBnaJkGVgLqb6S_FSKzKs0w47hgNT7LQIoxfeyJYohv GPxvBSJLWLZRolgnWTORc3nBCvqFslreQZLxMWy2Wfh0U2.wgvmG0rrg5kjT iUsMaS2WVidpIzko8yhgwUdmYM0zbcdPsmFw_hP95CFYd4g7IwIBUvsyWoFK Fp_zuB.kjGeo619cohVeZDh8VOZTkm8r5CCRC9mg0RhuS0drNP9.sylE8lgu Uf3jb2rpt2OXQ9F3of67tOap_lj9EegTA7r5rMdXPITBT7PQ039dUs3BdkLQ Q1y7mDvquYkV4eWE.JpJ4Ok.V4s7jPbH5YFPmezkPQguJ2aICNvgleimI0Mi gr01gI70tIU9rHnDmpKxYQikwyqb2A2RiUC0_W6SPMhdyp3O4epo2nyY7fJE sP6LrjcWCgzqKqyea8wp5gjTy Received: from [69.77.180.18] by web162001.mail.bf1.yahoo.com via HTTP; Mon, 03 Nov 2014 13:19:51 PST X-Rocket-MIMEInfo: 002.001,SGkgSGFyaSwKClllcywgSSBtYW5hZ2VkIHRvIGdldCB0aGUgdG9vbCB0byBhY3R1YWxseSBkbyBzb21ldGhpbmc6CgpqYXZhIC1jcCAiL3Vzci9saWIvZmx1bWUtbmcvbGliLyoiIG9yZy5hcGFjaGUuZmx1bWUudG9vbHMuRmx1bWVUb29sc01haW4gRkNJTlRFR1JJVFlUT09MIC1sIC9wYXRoL3RvLy5mbHVtZS9maWxlLWNoYW5uZWwvZGF0YQoKYnV0IGl0IGxvb2tzIGxpa2UgaXQganVzdCBnaXZlcyBtZSB0aGUgc2FtZSBleGNlcHRpb24gdGhhdCBGbHVtZSBpdHNlbGYgdGhyb3dzIHdoZW4gdHJ5aW5nIHRvIHIBMAEBAQE- X-Mailer: YahooMailWebService/0.8.203.733 References: <1415030959.73936.YahooMailNeo@web162004.mail.bf1.yahoo.com> <1415041386253.ef8fea7e@Nodemailer> Message-ID: <1415049591.49321.YahooMailNeo@web162001.mail.bf1.yahoo.com> Date: Mon, 3 Nov 2014 13:19:51 -0800 From: Andy Yang Reply-To: Andy Yang Subject: Re: FlumeTools/FileChannelIntegrityTool To: "user@flume.apache.org" In-Reply-To: <1415041386253.ef8fea7e@Nodemailer> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="1969517296-894647211-1415049591=:49321" X-Virus-Checked: Checked by ClamAV on apache.org --1969517296-894647211-1415049591=:49321 Content-Type: text/plain; charset=us-ascii Hi Hari, Yes, I managed to get the tool to actually do something: java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data but it looks like it just gives me the same exception that Flume itself throws when trying to restart: Exception in thread "main" java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598 at org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:628) at org.apache.flume.tools.FileChannelIntegrityTool.run(FileChannelIntegrityTool.java:87) at org.apache.flume.tools.FlumeToolsMain.run(FlumeToolsMain.java:64) at org.apache.flume.tools.FlumeToolsMain.main(FlumeToolsMain.java:31) Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length. at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49) at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:711) at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:776) at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:294) at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:484) at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:461) at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:579) at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:280) at com.google.protobuf.CodedInputStream.readGroup(CodedInputStream.java:240) at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:488) at com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:438) at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3444) at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3300) at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:212) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746) at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238) at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282) at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760) at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288) at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752) at org.apache.flume.channel.file.proto.ProtosFactory$Put.parseDelimitedFrom(ProtosFactory.java:3257) at org.apache.flume.channel.file.Put.readProtos(Put.java:98) at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:204) at org.apache.flume.channel.file.LogFileV3$SequentialReader.doNext(LogFileV3.java:359) at org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:624) ... 3 more So, I guess "Unable to read next transaction" due to "While parsing a protocol message, the input ended unexpectedly in the middle of a field" is not the kind of file channel integrity problem the tool can fix. Which leads me to ask, "is there anything else we can do with this file channel beyond discarding all the data currently persisted within?" Thanks, Andy On Monday, November 3, 2014 2:05 PM, Hari Shreedharan wrote: You need to specify only the data directories(comma-separated), the channel will look at all the files in the directory. Thanks, Hari On Mon, Nov 3, 2014 at 8:11 AM, Andy Yang wrote: Has anyone ever used the File Channel Integrity Tool released as part of Flume 1.4.0? There is absolutely no documentation on it but an outstanding JIRA issue from last year to write some. > >I've been poking it with a stick as follows: > >java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data/log-xxx > >but it returns far too quickly with no output, and no apparent change in the data file. > >I'm hoping it will help with a corrupt file channel error: > >java.lang.IllegalStateException: Channel closed [channel=c_xxx]. Due to java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598 >... >Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field. This could mean either than the input has been truncated or that an embedded message misreported its own length. > > >Thanks, >Andy > > > --1969517296-894647211-1415049591=:49321 Content-Type: text/html; charset=us-ascii
Hi Hari,

Yes, I managed to get the tool to actually do something:

java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data

but it looks like it just gives me the same exception that Flume itself throws when trying to restart:

Exception in thread "main" java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598
    at org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:628)
    at org.apache.flume.tools.FileChannelIntegrityTool.run(FileChannelIntegrityTool.java:87)
    at org.apache.flume.tools.FlumeToolsMain.run(FlumeToolsMain.java:64)
    at org.apache.flume.tools.FlumeToolsMain.main(FlumeToolsMain.java:31)
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either than the input has been truncated or that an embedded message misreported its own length.
    at com.google.protobuf.InvalidProtocolBufferException.truncatedMessage(InvalidProtocolBufferException.java:49)
    at com.google.protobuf.CodedInputStream.refillBuffer(CodedInputStream.java:711)
    at com.google.protobuf.CodedInputStream.readRawBytes(CodedInputStream.java:776)
    at com.google.protobuf.CodedInputStream.readBytes(CodedInputStream.java:294)
    at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:484)
    at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:461)
    at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:579)
    at com.google.protobuf.UnknownFieldSet$Builder.mergeFrom(UnknownFieldSet.java:280)
    at com.google.protobuf.CodedInputStream.readGroup(CodedInputStream.java:240)
    at com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:488)
    at com.google.protobuf.GeneratedMessage$Builder.parseUnknownField(GeneratedMessage.java:438)
    at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3444)
    at org.apache.flume.channel.file.proto.ProtosFactory$Put$Builder.mergeFrom(ProtosFactory.java:3300)
    at com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:212)
    at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:746)
    at com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:238)
    at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:282)
    at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:760)
    at com.google.protobuf.AbstractMessageLite$Builder.mergeDelimitedFrom(AbstractMessageLite.java:288)
    at com.google.protobuf.AbstractMessage$Builder.mergeDelimitedFrom(AbstractMessage.java:752)
    at org.apache.flume.channel.file.proto.ProtosFactory$Put.parseDelimitedFrom(ProtosFactory.java:3257)
    at org.apache.flume.channel.file.Put.readProtos(Put.java:98)
    at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:204)
    at org.apache.flume.channel.file.LogFileV3$SequentialReader.doNext(LogFileV3.java:359)
    at org.apache.flume.channel.file.LogFile$SequentialReader.next(LogFile.java:624)
    ... 3 more

So, I guess "Unable to read next transaction" due to "While parsing a protocol message, the input ended unexpectedly in the middle of a field" is not the kind of file channel integrity problem the tool can fix.

Which leads me to ask, "is there anything else we can do with this file channel beyond discarding all the data currently persisted within?"

Thanks,
Andy



On Monday, November 3, 2014 2:05 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:


You need to specify only the data directories(comma-separated), the channel will look at all the files  in the directory.


Thanks,
Hari


On Mon, Nov 3, 2014 at 8:11 AM, Andy Yang <ayang_ml@yahoo.ca> wrote:
Has anyone ever used the File Channel Integrity Tool released as part of Flume 1.4.0? There is absolutely no documentation on it but an outstanding JIRA issue from last year to write some. 
 
I've been poking it with a stick as follows:
 
java -cp "/usr/lib/flume-ng/lib/*" org.apache.flume.tools.FlumeToolsMain FCINTEGRITYTOOL -l /path/to/.flume/file-channel/data/log-xxx
 
but it returns far too quickly with no output, and no apparent change in the data file.
 
I'm hoping it will help with a corrupt file channel error:
 
java.lang.IllegalStateException: Channel closed [channel=c_xxx]. Due to java.io.IOException: Unable to read next Transaction from log file /path/to/.flume/file-channel/data/log-xxx at offset 513457598
...
Caused by: com.google.protobuf.InvalidProtocolBufferException: While parsing a protocol message, the input ended unexpectedly in the middle of a field.  This could mean either than the input has been truncated or that an embedded message misreported its own length.

Thanks,
Andy




--1969517296-894647211-1415049591=:49321--