Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 093DD200C17 for ; Fri, 10 Feb 2017 11:02:49 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 07E34160B5C; Fri, 10 Feb 2017 10:02:49 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 2DA04160B5B for ; Fri, 10 Feb 2017 11:02:48 +0100 (CET) Received: (qmail 96648 invoked by uid 500); 10 Feb 2017 10:02:47 -0000 Mailing-List: contact dev-help@flume.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flume.apache.org Delivered-To: mailing list dev@flume.apache.org Received: (qmail 96632 invoked by uid 99); 10 Feb 2017 10:02:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Feb 2017 10:02:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D4E8D185F63 for ; Fri, 10 Feb 2017 10:02:46 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.199 X-Spam-Level: X-Spam-Status: No, score=-1.199 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id PreWoCqUOPmW for ; Fri, 10 Feb 2017 10:02:45 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 0F8B65F576 for ; Fri, 10 Feb 2017 10:02:45 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 4D53DE04A8 for ; Fri, 10 Feb 2017 10:02:42 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id B060321D68 for ; Fri, 10 Feb 2017 10:02:41 +0000 (UTC) Date: Fri, 10 Feb 2017 10:02:41 +0000 (UTC) From: "Pan Yuxuan (JIRA)" To: dev@flume.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (FLUME-3054) hflushOrSync method in HDFS Sink should not treat ClosedChannelException as an error MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 10 Feb 2017 10:02:49 -0000 [ https://issues.apache.org/jira/browse/FLUME-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pan Yuxuan updated FLUME-3054: ------------------------------ Attachment: FLUME-3054.0001.patch Attach the initial patch. > hflushOrSync method in HDFS Sink should not treat ClosedChannelException as an error > ------------------------------------------------------------------------------------- > > Key: FLUME-3054 > URL: https://issues.apache.org/jira/browse/FLUME-3054 > Project: Flume > Issue Type: Bug > Components: Sinks+Sources > Reporter: Pan Yuxuan > Fix For: v1.8.0 > > Attachments: FLUME-3054.0001.patch > > > When use HDFS Sink in multiple threads, we face the error in log as below: > {code} > 09 Feb 2017 13:44:14,721 ERROR [hdfs-hsProt6-call-runner-4] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync:267) - Error while trying to hflushOrSync! > 09 Feb 2017 14:54:48,271 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:98) - Unexpected error while checking replication factor > java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:165) > at org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:84) > at org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:583) > at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:518) > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:418) > at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.nio.channels.ClosedChannelException > at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665) > at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2151) > at org.apache.hadoop.hdfs.DFSOutputStream.getNumCurrentReplicas(DFSOutputStream.java:2140) > ... 11 more > 09 Feb 2017 14:54:48,277 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:459) - process failed > org.apache.flume.auth.SecurityException: Privileged action failed > at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:49) > at org.apache.flume.auth.KerberosAuthenticator.execute(KerberosAuthenticator.java:63) > at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.nio.channels.ClosedChannelException > at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104) > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58) > at java.io.DataOutputStream.write(DataOutputStream.java:107) > at java.io.FilterOutputStream.write(FilterOutputStream.java:97) > at org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71) > at org.apache.flume.sink.hdfs.HDFSDataStream.append(HDFSDataStream.java:124) > at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550) > at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547) > at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1858) > at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:47) > ... 6 more > 09 Feb 2017 14:54:48,280 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160) - Unable to deliver event. Exception follows. > org.apache.flume.EventDeliveryException: org.apache.flume.auth.SecurityException: Privileged action failed > at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463) > at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) > at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.flume.auth.SecurityException: Privileged action failed > at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:49) > at org.apache.flume.auth.KerberosAuthenticator.execute(KerberosAuthenticator.java:63) > at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ... 1 more > Caused by: java.nio.channels.ClosedChannelException > at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665) > at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104) > at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58) > at java.io.DataOutputStream.write(DataOutputStream.java:107) > at java.io.FilterOutputStream.write(FilterOutputStream.java:97) > at org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71) > at org.apache.flume.sink.hdfs.HDFSDataStream.append(HDFSDataStream.java:124) > at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550) > at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547) > at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1858) > at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:47) > ... 6 more > {code} > The error shows that when call hflushOrSync(), the DataOutputStream was closed by other threads, so DataOutputStream throws ClosedChannelException. But the hdfs sink was not affected by the exception, DataOutputStream just throws the ClosedChannelException as a remind. So we should ignore the ClosedChannelException instead by logging as an error. -- This message was sent by Atlassian JIRA (v6.3.15#6346)