flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pan Yuxuan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLUME-3054) hflushOrSync method in HDFS Sink should not treat ClosedChannelException as an error
Date Fri, 10 Feb 2017 10:02:41 GMT

     [ https://issues.apache.org/jira/browse/FLUME-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pan Yuxuan updated FLUME-3054:
------------------------------
    Attachment: FLUME-3054.0001.patch

Attach the initial patch.

> hflushOrSync method in HDFS Sink should not treat ClosedChannelException as an error

> -------------------------------------------------------------------------------------
>
>                 Key: FLUME-3054
>                 URL: https://issues.apache.org/jira/browse/FLUME-3054
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>            Reporter: Pan Yuxuan
>             Fix For: v1.8.0
>
>         Attachments: FLUME-3054.0001.patch
>
>
> When use HDFS Sink in multiple threads, we face the error in log as below:
> {code}
> 09 Feb 2017 13:44:14,721 ERROR [hdfs-hsProt6-call-runner-4] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync:267)
 - Error while trying to hflushOrSync!
> 09 Feb 2017 14:54:48,271 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated:98)
 - Unexpected error while checking replication factor
> java.lang.reflect.InvocationTargetException
> 	at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.flume.sink.hdfs.AbstractHDFSWriter.getNumCurrentReplicas(AbstractHDFSWriter.java:165)
> 	at org.apache.flume.sink.hdfs.AbstractHDFSWriter.isUnderReplicated(AbstractHDFSWriter.java:84)
> 	at org.apache.flume.sink.hdfs.BucketWriter.shouldRotate(BucketWriter.java:583)
> 	at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:518)
> 	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:418)
> 	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> 	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedChannelException
> 	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665)
> 	at org.apache.hadoop.hdfs.DFSOutputStream.getCurrentBlockReplication(DFSOutputStream.java:2151)
> 	at org.apache.hadoop.hdfs.DFSOutputStream.getNumCurrentReplicas(DFSOutputStream.java:2140)
> 	... 11 more
> 09 Feb 2017 14:54:48,277 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hdfs.HDFSEventSink.process:459)
 - process failed
> org.apache.flume.auth.SecurityException: Privileged action failed
> 	at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:49)
> 	at org.apache.flume.auth.KerberosAuthenticator.execute(KerberosAuthenticator.java:63)
> 	at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedChannelException
> 	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665)
> 	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
> 	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:107)
> 	at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> 	at org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
> 	at org.apache.flume.sink.hdfs.HDFSDataStream.append(HDFSDataStream.java:124)
> 	at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550)
> 	at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547)
> 	at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1858)
> 	at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:47)
> 	... 6 more
> 09 Feb 2017 14:54:48,280 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.SinkRunner$PollingRunner.run:160)
 - Unable to deliver event. Exception follows.
> org.apache.flume.EventDeliveryException: org.apache.flume.auth.SecurityException: Privileged
action failed
> 	at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:463)
> 	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
> 	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.flume.auth.SecurityException: Privileged action failed
> 	at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:49)
> 	at org.apache.flume.auth.KerberosAuthenticator.execute(KerberosAuthenticator.java:63)
> 	at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	... 1 more
> Caused by: java.nio.channels.ClosedChannelException
> 	at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1665)
> 	at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:104)
> 	at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:107)
> 	at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> 	at org.apache.flume.serialization.BodyTextEventSerializer.write(BodyTextEventSerializer.java:71)
> 	at org.apache.flume.sink.hdfs.HDFSDataStream.append(HDFSDataStream.java:124)
> 	at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:550)
> 	at org.apache.flume.sink.hdfs.BucketWriter$7.call(BucketWriter.java:547)
> 	at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1858)
> 	at org.apache.flume.auth.UGIExecutor.execute(UGIExecutor.java:47)
> 	... 6 more
> {code}
> The error shows that when call hflushOrSync(), the DataOutputStream was closed by other
threads, so DataOutputStream throws ClosedChannelException. But the hdfs sink was not affected
by the exception, DataOutputStream just throws the ClosedChannelException as a remind. So
we should ignore the ClosedChannelException instead by logging as an error.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message